Trust & security

Should you block AI agents? Training bots vs search bots vs shopping agents

Listen · 8 min

Narrated by an AI voice. Code examples are on the page.

Short answer

Not all of them, and not in the same way. Blocking AI training crawlers such as GPTBot keeps your content out of model training without hurting search, according to OpenAI and Google. Blocking AI search crawlers can keep you out of AI answers. Shopping agents act for a real customer, may not follow robots.txt, and are better handled by verifying who they are and setting rules for what they may do.

Key takeaways

  • Training crawlers, search crawlers and agents need three different decisions.
  • Blocking GPTBot or Google-Extended does not remove you from search.
  • Verify shopping agents and set order rules instead of blocking them.

Should you block AI bots at all?

Some of them, maybe. The right answer depends on what each bot does, and AI bots now do three very different jobs.

"AI bots" covers three groups:

  1. Training crawlers collect pages that may be used to train AI models. GPTBot is the best-known example.
  2. AI search crawlers index pages so an AI product can cite and link to them in answers. OAI-SearchBot and PerplexityBot are examples.
  3. Agents and user-triggered fetchers visit because a person asked, right now. They read a page, compare products, or fill in a checkout for that person. ChatGPT-User, Perplexity-User, Google-Agent and ChatGPT agent fall here. See our user-triggered fetcher entry.

Each group gives you something different and costs you something different, so Block or allow is really three decisions. The sections below take them one at a time. For the exact robots.txt lines, use our robots.txt guide for AI agents.

Should you block AI training crawlers?

That is a business choice about your content, and blocking them does not cost you search traffic, according to the vendors.

OpenAI says that "Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models," and that its settings are independent: "Each setting is independent of the others." You can disallow GPTBot and still appear in ChatGPT search through OAI-SearchBot.

Google's equivalent is Google-Extended, a robots.txt token that controls whether your content may be used for training Gemini models and for grounding. Google says "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Reasons store owners block training crawlers:

  • They do not want product copy or photos reused in model training.
  • They want to reduce crawl load.

Reasons some do not bother:

  • Their product pages are already widely syndicated through feeds and marketplaces.
  • They want models to know their brand.

Either choice is reasonable. If you want to state your preference more precisely, Cloudflare's Content Signals Policy adds a robots.txt line such as Content-Signal: search=yes, ai-train=no. Cloudflare is clear that signals "express preferences; they are not technical countermeasures against scraping."

Should you block AI search crawlers?

Usually not, if you want to be found. Blocking them can keep your pages out of AI answers and the links in them.

OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links. Perplexity says PerplexityBot is designed to surface and link websites in search results and "is not used to crawl content for AI foundation models."

For most stores, being cited when a shopper asks an assistant "where can I buy this?" is worth more than the crawl cost. Block a search crawler only if you have a specific reason, such as content you sell by subscription.

Should you block AI shopping agents?

In most cases, no. Verify them and set rules instead. A shopping agent is acting for a real person, often one who is ready to buy.

Two facts make blocking agents a poor fit:

  • robots.txt may not apply. Google says its user-triggered fetchers "generally ignore robots.txt rules" because a user asked for the fetch. Perplexity says the same of Perplexity-User. OpenAI says that for ChatGPT-User, "robots.txt rules may not apply."
  • Some agents look like normal browsers. Agents that run inside a person's own browser may not announce themselves at all, so a blocklist cannot catch them reliably.

Cloudflare's own setup reflects this. Its AI Crawler and Agent access policies let you choose an action for each of three categories, Search, Agent and Training. Its documentation says that for new domains all three default to "Allow (do not block)," and that "Cloudflare recommends Allow (do not block) for Personal Agents."

What works better than blocking:

  1. Verify identity. A request that only claims to be an agent is a claim. Agents that sign their requests with Web Bot Auth can be verified, and Cloudflare now includes signed agents in its Verified category, with a Direct or Intermediary label. See Cloudflare verified bots and signed agents.
  2. Treat failed checks as ordinary visitors. Requests that claim to be a known agent but fail verification should get no special treatment. Our guide to spoofed AI crawlers explains the checks.
  3. Limit actions, not visits. Let agents read freely. Put rules on what they can do: order limits, rate limits, and a person reviewing large or unusual orders.

When does blocking an agent make sense?

When a specific visitor causes harm, not because it is an AI.

Good reasons to block or rate-limit a particular agent or address:

  • it sends far more requests than any real shopping session would
  • it probes for admin pages or secret files
  • it repeatedly fails verification while using a well-known agent's name
  • it places orders you have to cancel

Before you block, check whether the visitor really is who it claims to be. A request that claims to be a well-known agent but fails verification has not shown that it comes from that agent, and blocking that request does not block the agent's verified requests. A request that passes verification has shown it comes from the named operator, so if it misbehaves, a rate limit or a narrow rule on that one agent is usually enough, and you can contact the operator through the documentation it publishes.

Block narrowly and by evidence. A blanket block on "AI" also blocks customers who have chosen to shop through an assistant, at a time when AI products are adding ways to buy, such as Microsoft's Copilot Checkout.

How Sill helps

Sill shows you which AI agents and bots visit your site, the pages they read, and what they ask for and buy, so you can decide with evidence. It verifies signed agent requests (Web Bot Auth) on Cloudflare and WordPress installs. When an AI agent places an order, your rules approve it, decline it or hold it for a person to review, and every decision is signed and recorded. Run a free readiness scan to check your AI access settings.

Questions people ask

Will blocking GPTBot remove my site from ChatGPT search?

No. OpenAI says its robots.txt settings are independent, so you can disallow GPTBot and still allow OAI-SearchBot, which is the crawler behind ChatGPT search results.

Does blocking Google-Extended hurt my Google rankings?

Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls use of your content for Gemini training and grounding.

Can I block AI agents completely?

You can block known user agents and IP ranges at your server or CDN, but agents that run in a person's own browser can look like ordinary visitors. Blocking also turns away customers who shop through agents.

Is it legal for AI agents to ignore robots.txt?

robots.txt is a voluntary convention, and vendors such as Google and Perplexity document that their user-triggered fetchers generally ignore it. Whether that matters legally depends on your jurisdiction and terms. This is not legal advice.

Sources

  1. OpenAI: Overview of OpenAI crawlers · checked 16 Nov 2026
  2. Google: Google's common crawlers · checked 16 Nov 2026
  3. Google: Google's user-triggered fetchers · checked 16 Nov 2026
  4. Perplexity: Perplexity crawlers · checked 16 Nov 2026
  5. Cloudflare docs: AI Crawler and Agent access policies · checked 16 Nov 2026
  6. Cloudflare docs: Verified bots and signed agents · checked 16 Nov 2026
  7. Microsoft Advertising: Copilot Checkout and Brand Agents · checked 16 Nov 2026
  8. Cloudflare blog: Giving users choice with Cloudflare's new Content Signals Policy · checked 16 Nov 2026

Written by The Sill Signals Team. Last reviewed 16 Nov 2026.

Share this articleLinkedInXBlueskyEmail

Keep reading

See which AI agents visit your site, and decide what they may do.

Scan your site free