Content-Signal
Content-Signal in robots.txt: ai-train, ai-input and search explained
How the Content-Signal line in robots.txt works: ai-train, ai-input and search, Cloudflare's defaults, the IETF aipref drafts, and who honors them.
AI training crawler
ClaudeBot is Anthropic's crawler. Anthropic says it collects web content that could help train its generative AI models.
Anthropic says its bots honor industry-standard robots.txt directives and also supports the non-standard Crawl-delay.
User-agent: ClaudeBot Allow: /
Sill Agent Intelligence
Sill Agent Intelligence is building ClaudeBot's behaviour profile from verified traffic across the Sill network: reach, identity checks, activity trend and the pages it targets. It appears here automatically when ready.
Content-Signal
How the Content-Signal line in robots.txt works: ai-train, ai-input and search, Cloudflare's defaults, the IETF aipref drafts, and who honors them.
reverse DNS
Verify AI bots by IP range and reverse DNS: the official IP lists and DNS checks for OpenAI, Anthropic, Google, Perplexity, Bing, Apple, Amazon and more.
Agent identity
AI agent identity explained for website owners: user agents, IP lists, reverse DNS, signed requests and verified bots, and how far to trust each one.
Block it if you do not want future content used to train Anthropic models. Anthropic says this signals that your future content should be excluded from training datasets.
Anthropic warns this may not reliably opt you out, because it stops the bot from reading your robots.txt. Use a robots.txt rule instead.