Short answer
If you want to be found and bought from but not used for AI training, allow the search and answer bots (such as OAI-SearchBot, PerplexityBot and Claude-SearchBot) and block the training crawlers (such as GPTBot, ClaudeBot and CCBot). User-initiated fetchers like ChatGPT-User may not follow robots.txt at all, so treat the file as a request, not a lock.
Key takeaways
- AI bots fall into three kinds: training, search, and user-initiated.
- Block training bots to opt out of training without leaving AI search.
- robots.txt is a request; some user-initiated fetchers may ignore it.
What are the three kinds of AI bot?
AI bots do three different jobs, and each one deserves its own rule. The three kinds are training crawlers, search and answer crawlers, and user-initiated fetchers.
Training crawlers collect pages to help build AI models. Examples are GPTBot (OpenAI), ClaudeBot (Anthropic) and CCBot (Common Crawl). Blocking them tells the company not to use your content for training.
Search and answer crawlers build the index an AI product searches when someone asks a question. Examples are OAI-SearchBot (OpenAI), PerplexityBot (Perplexity) and Claude-SearchBot (Anthropic). Block these and your store is less likely to be shown or linked in AI answers.
User-initiated fetchers visit one page because a person asked. Examples are ChatGPT-User, Perplexity-User and Claude-User. They are not crawling your whole site. They are reading a page on behalf of a shopper or researcher.
Here is how the vendors describe their own bots:
| Bot | Company | What it does |
|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train its models |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search features |
| ChatGPT-User | OpenAI | User-initiated visits from ChatGPT and custom GPTs |
| ClaudeBot | Anthropic | Collects content that could contribute to training |
| Claude-SearchBot | Anthropic | Improves search result quality |
| Claude-User | Anthropic | Fetches pages when a Claude user asks |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search |
| Perplexity-User | Perplexity | Fetches pages when a user asks a question |
Read more about the main ones: GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot. Each bot announces itself with a user agent string, and that string is what robots.txt rules match.
Which AI bots might ignore robots.txt?
Two vendors say so plainly for their user-initiated fetchers. Others make no such statement, so check each vendor's page.
OpenAI says ChatGPT-User handles actions started by a user, and that "because these actions are initiated by a user, robots.txt rules may not apply."
Perplexity says that because a user requested the fetch, Perplexity-User "generally ignores robots.txt rules."
Anthropic takes a different position. Its help page says its bots honor industry-standard robots.txt directives, and that includes Claude-User.
The practical point: a robots.txt rule for a user-initiated fetcher may do nothing. If you need to control those visits, use your firewall, your CDN or your site's own rules. Do not rely on robots.txt alone.
What are Google-Extended and Applebot-Extended?
They are control tokens, not bots. You can write rules for them in robots.txt, but they never show up in your server logs as a visitor.
Google says Google-Extended "doesn't have a separate HTTP request user agent string." Crawling is done by Google's existing crawlers. The token lets you decide whether what Google crawls may be used for training future Gemini models and for grounding in Gemini Apps. Google also says it does not affect inclusion in Google Search and is not a ranking signal.
Apple says "Applebot-Extended does not crawl webpages." It works as a secondary setting on top of Applebot. Disallowing it opts your content out of training Apple's generative models. Apple says pages that disallow it can still appear in search results.
So if you check your logs for Google-Extended or Applebot-Extended, you will find nothing. That is normal.
What should a store's robots.txt look like?
Here is one option for a store that wants to be found and bought from, but not used for training. It is a starting point, not the only correct answer. Your legal, brand and licensing needs may differ.
# Training crawlers: opt out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
# Control-only tokens: opt out of model training
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Search and answer crawlers: allow, so you can be found
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# User-initiated fetchers: allow
User-agent: ChatGPT-User
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-User
Allow: /
# Everyone else
User-agent: *
Disallow: /cart
Disallow: /checkout
Allow: /
A few notes on this file.
- Keep carts, checkouts and account pages out of reach for general crawlers. Add your own private paths.
- An empty
Allowfor a named bot is a clear statement of intent, even though allowing is the default. - Put the file at the top level of each domain and subdomain. Anthropic says to do this for each subdomain.
- Changes are not instant. OpenAI says its systems can take about 24 hours to adjust after you update the file.
- New bots appear often. Review the file every few months against each vendor's current list.
If you want to be found in AI search, do not block the search bots. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers.
What is the Content Signals line?
It is a newer, optional line that states your preferences by purpose instead of by bot name. Cloudflare introduced it as its Content Signals Policy. The line goes in robots.txt and looks like this:
User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /
There are three signals, each set to yes or no:
- search: building a search index and providing search results. Cloudflare says this does not include AI-generated search summaries.
- ai-input: using content as input to AI models, for example for grounding or retrieval.
- ai-train: training or fine-tuning AI models.
If you leave a signal out, you have not expressed a preference for it. Cloudflare is clear that content signals "express preferences; they are not technical countermeasures against scraping," and that some companies might ignore them. Support among AI companies is not universal, so use the line alongside the named rules above, not instead of them.
Why is robots.txt a request, not a lock?
Because nothing enforces it. A well-behaved bot reads the file and follows it. The file cannot stop a bot that chooses not to read it, and it cannot stop a request that pretends to be a bot it is not.
That matters in two ways.
First, as covered above, some user-initiated fetchers say outright that they may not follow it.
Second, anyone can put "ChatGPT-User" or "GPTBot" in a request header. A rule that names a bot cannot tell the real one from a copy. Anthropic itself advises against blocking by IP address as a way to opt out, since that "may not work correctly or persistently guarantee an opt-out." To check who is really knocking, see how to tell if a visitor is really ChatGPT.
Note. Use robots.txt to state your policy. Use verification and your firewall to enforce it.
How Sill helps
Sill shows you which AI agents and bots visit your site and which pages they read, so you can see whether your robots.txt choices match what actually happens. It also publishes the files agents read, such as llms.txt, and keeps them current. A free readiness scan at /scan needs no account.
Questions people ask
Does blocking GPTBot remove my store from ChatGPT?
No. OpenAI says disallowing GPTBot signals that your content should not be used to train its models. Appearing in ChatGPT search answers is controlled by a different bot, OAI-SearchBot.
Do I need a separate robots.txt for each subdomain?
Yes. Anthropic says to add its rules to the robots.txt in the top-level directory of each subdomain. Treat any other crawler the same way.
How long do robots.txt changes take to work?
OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust. Other companies do not all publish a timeframe.
Does blocking Google-Extended hurt my Google Search ranking?
Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
Sources
- Common Crawl: CCBot · checked 15 Nov 2026
- OpenAI: Overview of OpenAI crawlers · checked 15 Nov 2026
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? · checked 15 Nov 2026
- Perplexity: Perplexity crawlers · checked 15 Nov 2026
- Google Search Central: Google's common crawlers · checked 15 Nov 2026
- Apple: About Applebot · checked 15 Nov 2026
- Cloudflare: Giving users choice with Cloudflare's new Content Signals Policy · checked 15 Nov 2026