Robots.txt AI Crawler Blocker: Stop GPTBot, ClaudeBot, Google-Extended, Perplexity & 17 Other AI Bots

What an AI crawler blocker does for your site

Generative AI systems collect web content in two distinct ways. Training crawlers, like OpenAI’s GPTBot, Anthropic’s ClaudeBot, Google-Extended and Common Crawl’s CCBot, sweep the open web to build the datasets that future model versions will learn from. In-product agents, like ChatGPT-User, Perplexity-User and Claude’s web fetcher, do live retrieval when a chatbot needs an up-to-date page to answer a user prompt. Each kind of bot has its own user-agent string and its own purpose, and many of them now honour robots.txt as the standard opt-out signal.

This generator builds a robots.txt file that addresses every documented AI bot we know about as of 2026: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, claude-web, anthropic-ai, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, Meta-ExternalAgent, FacebookBot, Bytespider, CCBot, Amazonbot, Diffbot, omgilibot, YouBot, Kagibot, Cohere-AI, Timpibot and a few legacy variants. You toggle each bot individually or use a preset like “Block AI training” or “EU AIA Article 4(3) opt-out”. The tool keeps Googlebot, Bingbot and other search crawlers untouched by default so your organic visibility stays intact.

How AI crawler blocking with robots.txt works

The Robots Exclusion Standard, originally drafted in 1994 and formalised in RFC 9309 in 2022, defines a plain-text file at /robots.txt on the root of a domain. The file lists one or more User-agent blocks, each followed by Allow and Disallow directives. When a crawler arrives, it fetches the file, finds the block that matches its user-agent name, and obeys the rules. The mechanism is voluntary: there is no technical enforcement, only a public convention that reputable crawler operators follow. The largest AI vendors publish their bot names and have committed to honouring robots.txt.

1. Pick the bots you want to block. Training crawlers are the obvious target if you want to opt out of being used as model fuel. In-product fetchers matter if you do not want your live pages summarised by other people’s chatbots.
2. Decide on the path scope. Disallow: / blocks the entire site for that user-agent. You can also block sub-paths only, for example /blog/ or /archive/, and leave the rest crawlable.
3. Keep search engines crawling by not adding their user-agents to the block list. Googlebot, Bingbot, DuckDuckBot, and others should remain explicitly or implicitly allowed.
4. Add server-level enforcement when the legal or commercial stakes are high. The Nginx and Apache blocks generated by this tool reject the listed user-agents with HTTP 403 even if they ignore robots.txt.
5. Deploy and verify by uploading the file to your web root and fetching https://yourdomain.com/robots.txt. The tool’s verification panel suggests the exact lines to check.

Common use cases for blocking AI crawlers

• Publisher protecting editorial work. If your business model depends on visits to your articles, having a language model that summarises everything you publish without sending traffic back is a long-term threat. Blocking training crawlers is the cleanest signal that your content is not free training material.
• SaaS hiding paid documentation. Knowledge bases that sit behind a paywall or a login should not show up in scraped training corpora. Blocking GPTBot and CCBot reduces the risk that paying customers’ answers leak into a public model.
• EU rightholder exercising Article 4(3) opt-out. The EU AI Act extends the existing copyright text-and-data-mining opt-out into a machine-readable signal. The “EU AIA opt-out” preset on this page produces the user-agent blocks that match what major AI vendors have announced they will respect.
• Brand keeping AI answers consistent. Some brands prefer that AI products reference their authoritative help centre through search rather than caching their content. Blocking in-product fetchers, but keeping training crawlers off, is one signal that says “use my official channels”.
• Internal staging or low-quality content. A draft blog or staging environment should not leak into model training. Adding the AI bot block on top of a generic Disallow: / keeps things tidy even if the staging environment becomes public by accident.
• Compliance with internal policy. Some organisations require a documented opt-out signal as part of their data governance. Even if enforcement is imperfect, having the file is part of meeting the policy.

Limitations and privacy notes

Blocking AI crawlers with robots.txt is the right first step but it is not bulletproof. The file is a public request, not a technical barrier. Bots that ignore the standard, scrape through residential proxies, or download content via a non-AI intermediary (like Common Crawl on behalf of a downstream model) will still see your pages. Old training datasets that already contain your content are not affected by a future robots.txt change. Different AI vendors interpret the same user-agent name slightly differently, and some bots like Google-Extended only opt the site out of training without removing it from Google Search.

This tool runs in your browser. The user-agent list, the sitemap URL, the custom paths and the existing robots.txt you paste are processed locally. Nothing is sent to PeopleAreGeek or to any third party. The bot reference table is shipped with the page; we update it when vendors publish new user-agents.

Frequently asked questions

Related tools and resources

Robots.txt is one signal in a wider site posture. The tools below help validate the file, test crawler access, audit headers and check sitemap visibility.