AI Crawler Access Generator

AI Crawler Access Generator: Allow or Block GPTBot, ClaudeBot & More, Free, Right in Your Browser

A plain Disallow: / under User-agent: * blocks everything, including the search engines you still want crawling your site. Since 2023, a growing list of named AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Bytespider, and others) has made that approach too blunt: some of these bots crawl for AI model training, some crawl to power a search-style answer, and some only fetch a page the moment a real person asks a question that needs it. Treating all three the same way throws away the distinction most site owners actually care about. This tool lets you set a rule per bot, by name, using the real user-agent tokens each company has published.

Every bot listed here is a real, currently-documented AI crawler, not a placeholder. The names and purposes come from each company's own crawler documentation where one exists (OpenAI, Anthropic, Google, Perplexity, Common Crawl), or from independently corroborated reporting where a company hasn't published a dedicated page (ByteDance's Bytespider, Amazon's Amazonbot, Meta's meta-externalagent). See the sourcing note at the bottom of this page for the exact references used.

1Training, Search, and Answer: Why the Same Company Can Have Three Different Bots

OpenAI and Anthropic both split their crawling into three distinct bots with three distinct purposes, and this tool keeps that same split. A training bot, like GPTBot or ClaudeBot, crawls content that may end up as training data for a future model. A search bot, like OAI-SearchBot or PerplexityBot, crawls and indexes pages the way a traditional search engine does, to power a search-style feature inside the AI product. An answer bot, like ChatGPT-User, Claude-User, or Perplexity-User, doesn't crawl at all in the bulk sense: it fetches one specific page, in real time, only because a person's question to that assistant required visiting it. A common pattern among sites that want AI visibility without feeding model training is to block the training bots while leaving the search and answer bots allowed, since those are the ones that tend to cite or link back to the source page.

2Simple vs Pro

Simple Mode covers the common case: pick an Allow or Block stance for each bot and generate the rules. Pro Mode, currently free to use, is built for the messier real situation, where a site already has a robots.txt file, wants different rules for different folders, or wants to apply a sensible starting point and adjust from there instead of configuring 13 bots one at a time.

3Common Reasons Site Owners Set Rules for AI Crawlers

None of these need a developer on standby or a CMS plugin, just a clear decision per bot and a text file placed at your site's root. That's the gap this tool fills: an accurate, up-to-date list of real AI crawler names so the decision is about policy, not about guessing what to even type into robots.txt.

4Per-Bot Path Exceptions and Merging With an Existing robots.txt

A single site-wide Allow or Block per bot isn't always enough. Pro Mode's path exceptions let you set a general stance for a bot and then carve out specific folders that behave the opposite way, for example allowing ClaudeBot everywhere except a /private/ directory, or blocking GPTBot everywhere except a public /press/ section you want cited. The tool writes the correct Allow and Disallow lines for that combination automatically.

Most sites also already have a robots.txt file with rules for ordinary search engines, sitemaps, or crawl restrictions on admin folders. Pasting that file into the Combine card keeps every one of those existing lines exactly as written and appends the new AI-bot blocks underneath as their own User-agent sections. If a bot you configured is already named somewhere in the pasted file, the tool skips adding a second block for it rather than creating a conflicting duplicate.

5Frequently Asked Questions