AI training crawler

CCBot

CCBot is the crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data. Others, including AI developers, can use that archive.

Sources: Common Crawl's own documentation and the Sill agent registry · checked 15 Nov 2026

Company
Common Crawl
Acts for
Common Crawl, an open web archive
Follows robots.txt
Yes
How to verify
Published IP list, Reverse DNS

robots.txt

Common Crawl says to block CCBot by adding a User-agent: CCBot group with Disallow: /.

User-agent: CCBot
Allow: /

How to check it is really CCBot

How verification works →

Sill Agent Intelligence

What it does on websites PROFILE IN PROGRESS

Sill Agent Intelligence is building CCBot's behaviour profile from verified traffic across the Sill network: reach, identity checks, activity trend and the pages it targets. It appears here automatically when ready.

Read more on Signals

Questions about CCBot

Should I block CCBot?

Block it if you do not want your pages in the Common Crawl archive. Because others reuse the archive, this is also how some sites keep content out of later datasets.

How do I verify CCBot?

Common Crawl publishes IP ranges as JSON and says a forward and reverse DNS lookup should match a crawl.commoncrawl.org hostname. It warns that other crawlers sometimes falsely claim to be CCBot.

Sources

  1. Common Crawl: CCBot · checked 15 Nov 2026

Official documentation: https://commoncrawl.org/ccbot · Spotted an error? Tell us

See which AI agents visit your site, and decide what they may do.

Scan your site free