AI search crawler

Claude-SearchBot

Claude-SearchBot crawls the web to analyze content and improve the relevance and accuracy of search results for Claude users.

Sources: Anthropic's own documentation and the Sill agent registry · checked 15 Nov 2026

Company
Anthropic
Acts for
Anthropic, for search quality
Follows robots.txt
Yes
How to verify
Published IP list

robots.txt

Anthropic says its bots honor robots.txt, and that blocking Claude-SearchBot prevents indexing for search optimization.

User-agent: Claude-SearchBot
Allow: /

How to check it is really Claude-SearchBot

How verification works →

Sill Agent Intelligence

What it does on websites PROFILE IN PROGRESS

Sill Agent Intelligence is building Claude-SearchBot's behaviour profile from verified traffic across the Sill network: reach, identity checks, activity trend and the pages it targets. It appears here automatically when ready.

Other Anthropic agents

Read more on Signals

Questions about Claude-SearchBot

Should I block Claude-SearchBot?

Anthropic says blocking it may reduce your visibility and the accuracy of your content in user search results. Allow it if you want to be found through Claude.

Is it used for training?

Anthropic describes it as improving search results. Training is described under ClaudeBot.

Sources

  1. Anthropic: Does Anthropic crawl data from the web? · checked 15 Nov 2026

Official documentation: https://support.claude.com/en/articles/8896518 · Spotted an error? Tell us

See which AI agents visit your site, and decide what they may do.

Scan your site free