Definition
A training crawler is a bot that visits websites to collect content that may be used to train or fine-tune AI models. Examples include OpenAI's GPTBot and Anthropic's ClaudeBot. Operators publish a separate robots.txt token for training, so you can block it without blocking their search or user-triggered bots.
Why it matters
A training crawler is the kind of AI bot most site owners think of when they talk about blocking AI. It reads pages in bulk so the content may help build future models. It does not send you a shopper, and it does not answer a question someone just asked about your store.
That makes it a separate decision from search bots and shopping agents. Blocking a training crawler does not, by itself, remove you from AI search answers or stop an agent a customer sent to your site. Some owners block training and allow the rest. Others allow training because they want their brand to be known to future models. Either is a reasonable choice.
How it works
The major AI companies now split their bots by purpose and give each its own robots.txt token:
- OpenAI says GPTBot "is used to crawl content that may be used in training our generative AI foundation models." It adds: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." OpenAI says each of its settings is independent, so you can allow OAI-SearchBot for search while disallowing GPTBot.
- Anthropic says "ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training." It runs separate bots, Claude-User and Claude-SearchBot, for user requests and search.
- Google uses a token rather than a separate bot. Google-Extended is "a standalone product token" for managing whether content Google crawls "may be used for training future generations of Gemini models." Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
Cloudflare's bot classification follows the same split. Its list of bot behaviors includes Search, Agent and Training, and it defines Training as "Crawling to train or fine-tune models."
robots.txt is a request, not a lock: a crawler has to choose to follow it. A firewall rule is the way to enforce a block.
Example
You want to stay out of AI training but still appear in ChatGPT search. Your robots.txt says:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
GPTBot stops collecting your pages for training. OAI-SearchBot keeps indexing them for search. See the GPTBot, ClaudeBot and Google-Extended profiles, and should you block AI agents?
Sources
- OpenAI: Overview of OpenAI crawlers · checked 17 Nov 2026
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? · checked 17 Nov 2026
- Google: Google's common crawlers · checked 17 Nov 2026
- Cloudflare: Verified bots · checked 17 Nov 2026