Merchant playbooks

Content-Signal in robots.txt: ai-train, ai-input and search explained

Listen · 11 min

Narrated by an AI voice. Code examples are on the page.

Short answer

Content-Signal is a robots.txt line from Cloudflare that states how you want content used after it is fetched: search, ai-input (feeding AI answers) and ai-train (model training), each set to yes or no. It is a preference, not a block, and the AI crawler pages we checked do not say they follow it. Google and Anthropic have pointed instead to a separate IETF-based training signal, so use Content-Signal alongside named bot rules, not instead of them.

Key takeaways

  • Content-Signal sets search, ai-input and ai-train preferences in robots.txt.
  • It states a preference; it does not technically block any bot.
  • Google and Anthropic back a different, IETF-based training signal.

How do you write a Content-Signal line?

You add one Content-Signal line to a group in your robots.txt, listing up to three signals separated by commas, each set to yes or no. Cloudflare's own example looks like this:

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

A few rules of the syntax, as Cloudflare describes them:

  • Three keys. The signals are search, ai-input and ai-train.
  • Two values. Each is yes or no.
  • Leaving one out is allowed. Cloudflare says an omitted signal "neither grants nor restricts permission via content signal." It also says that does not mean you have no preference about that use. You just have not stated one in the file.
  • It sits inside a group. Cloudflare's examples put it under User-Agent: *, next to the usual Allow and Disallow lines, so it applies to every bot that reads that group.

Cloudflare also suggests a comment block above the line. It begins "As a condition of accessing this website, you agree to abide by the following content signals" and then defines each signal. The policy is released under a CC0 license, so you can copy it. For the basics of the file itself, see robots.txt for AI agents and the short content signals definition.

What do search, ai-input and ai-train each cover?

Each signal covers a different use of your pages after they are fetched. The difference between the last two is the one most site owners get wrong.

  • search means "building a search index and providing search results." Cloudflare says this covers returning links and short excerpts, but not AI-generated search summaries.
  • ai-input means "inputting content into one or more AI models." Cloudflare's examples are real-time uses such as retrieval-augmented generation and grounding for AI search answers. This is how an AI answer quotes your product page today.
  • ai-train means "training or fine-tuning AI models." This is about building future models, not answering a question now.

Here is how common goals map to the line:

Your goal Content-Signal line
Be found in search and quoted in AI answers, but not used for training search=yes, ai-input=yes, ai-train=no
Be found in search only, no AI use at all search=yes, ai-input=no, ai-train=no
No stated preference on AI answers (Cloudflare's managed default) search=yes, ai-train=no

Most stores that want AI shoppers to find them will not want ai-input=no. It asks AI tools not to use your pages when answering a shopper's question.

What does Cloudflare's managed robots.txt add?

If you turn on managed robots.txt in Cloudflare, it writes content signals for you. Cloudflare says the feature "is available on all plans."

According to Cloudflare's docs, the managed block includes:

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

The docs describe use=reference as "Index, excerpt, and link back." Note that this value is not one of the three signals in the original policy post. Depending on your bot settings, the block can also add Disallow rules for named AI crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended.

If you already have a robots.txt, Cloudflare says it will "prepend our managed robots.txt before your existing robots.txt" and serve both as one file. So check the combined result for rules that contradict each other. When it launched the policy in September 2025, Cloudflare said customers had turned on managed robots.txt for "over 3.8 million domains."

Cloudflare's policy text also describes restrictions expressed through content signals as express reservations of rights under EU copyright rules. Whether that has legal effect for your site is a question for your lawyer. This is not legal advice.

How is Content-Signal different from the IETF aipref drafts?

They aim at the same problem with different words and a different field name. Content-Signal is a Cloudflare policy. The IETF's aipref working group is drafting a shared standard, and at the time of writing it is still a draft.

Two working group drafts matter:

  • The vocabulary, draft-ietf-aipref-vocab-08, dated September 2026, defines three categories: train-ai (AI training), ai-use (AI use) and search. Each takes y or n. Its note to readers says its contents "DO NOT REFLECT CONSENSUS of the Working Group either in whole or part."
  • The attachment draft, draft-ietf-aipref-attach-05, dated August 2026, says how to put those preferences in robots.txt and HTTP headers. It uses a field called Content-Usage.

The attachment draft's example:

User-Agent: *
Allow: /
Content-Usage: train-ai=n
Purpose Cloudflare Content-Signal IETF aipref draft
Model training ai-train=no train-ai=n
AI use of content ai-input=no ai-use=n
Search search=yes search=y
Field name Content-Signal Content-Usage

The definitions do not line up exactly. For example, the vocabulary draft says its search category "overrides any usage that falls into other categories, including AI Training." Treat the two as related, not interchangeable.

Which AI companies honor content signals?

At the time of writing, none of the major AI crawler pages we checked say they follow Content-Signal. Cloudflare itself says "Some companies might simply ignore them."

Here is what each company's own documentation says:

  • OpenAI's crawler overview describes control through user-agent rules, such as disallowing GPTBot. It does not mention Content-Signal.
  • Perplexity's crawler page describes allowing or blocking PerplexityBot and Perplexity-User by name. It does not mention Content-Signal.
  • Google says in its crawler documentation that it intends to support "the AIPREF shared machine-readable training preference signal" in early 2027, using a __usage_ai-training user-agent token. In an October 30, 2026 post, Google said existing company-specific controls such as Google-Extended will keep working alongside the shared signal.
  • Anthropic's help center says that on October 30, 2026 it began recognizing a machine-readable signal for a no-AI-training preference, based on work from the IETF aipref working group, and that it is rolling this out across its services over time. It also says a robots.txt rule that names ClaudeBot takes priority.

That shared training signal is not Content-Signal. It is described in an individual Internet-Draft, draft-kelley-aipref-training-preference-signal, written by an author at Anthropic. The datatracker notes that an individual draft has no formal standing at the IETF. Its example uses an ordinary robots.txt group:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The draft explains that this states a preference against AI training everywhere except /press-kit/.

Should you add Content-Signal to your robots.txt?

Yes, if you want a clear public statement of your wishes, but do not rely on it alone. It costs nothing and can sit next to your other rules. It just will not do much by itself until AI companies say they read it.

A practical setup for a store that wants to be found but not trained on:

User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

User-Agent: __usage_ai-training
Disallow: /

User-Agent: GPTBot
Disallow: /

User-Agent: ClaudeBot
Disallow: /

User-Agent: Google-Extended
Disallow: /

Check this against your own goals. Blocking a training crawler by name is the method each company documents today, and the shared token is for the companies that support it or have said they plan to. None of these lines stop a bot that chooses to ignore robots.txt. For that you need firewall rules, which our guide to allowing good AI agents through Cloudflare covers, and a way to see who actually visits, covered in how to see AI agent traffic your analytics misses.

How Sill helps

The free readiness scan at /scan checks your robots.txt for AI access rules and content signals, and shows the exact URL it read. Once Sill is installed, it shows which AI agents and bots visit your site and the pages they read, so you can compare your stated preferences with what actually happens.

Questions people ask

Does Content-Signal: ai-train=no stop GPTBot?

Not on its own. OpenAI's crawler documentation describes control through user-agent rules such as disallowing GPTBot, and does not mention Content-Signal. Add a named GPTBot rule if you want to opt out of OpenAI training.

What happens if I leave ai-input out of the line?

Cloudflare says an omitted signal means the site operator neither grants nor restricts permission for that use through content signals. Its managed robots.txt leaves ai-input out by default.

Is Content-Signal an official standard?

No. It is a Cloudflare policy released under a CC0 license. The IETF's aipref working group is drafting its own vocabulary and a robots.txt field called Content-Usage, and those drafts are still works in progress.

Is a content signal legally binding?

Cloudflare's policy text calls the restrictions express reservations of rights, and Cloudflare says signals may carry legal weight in some jurisdictions. Whether that holds for your site depends on where you and the AI company are. This is not legal advice.

Sources

  1. Cloudflare blog: Giving users choice with Cloudflare's new Content Signals Policy · checked 17 Nov 2026
  2. Cloudflare docs: Managed robots.txt · checked 17 Nov 2026
  3. IETF: draft-ietf-aipref-vocab-08, A Vocabulary For Expressing AI Usage Preferences · checked 17 Nov 2026
  4. IETF datatracker: draft-ietf-aipref-attach-05 · checked 17 Nov 2026
  5. IETF: draft-kelley-aipref-training-preference-signal-01, Machine-Readable Training Preference Signal · checked 17 Nov 2026
  6. Google Search Central blog: Supporting a shared AI training control for websites (October 30, 2026) · checked 17 Nov 2026
  7. Google: Google's common crawlers · checked 17 Nov 2026
  8. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? · checked 17 Nov 2026
  9. OpenAI: Overview of OpenAI crawlers · checked 17 Nov 2026
  10. Perplexity: Perplexity crawlers · checked 17 Nov 2026

Written by The Sill Signals Team. Last reviewed 17 Nov 2026.

Share this articleLinkedInXBlueskyEmail

Keep reading

See which AI agents visit your site, and decide what they may do.

Scan your site free