reverse DNS
Verifying AI bots by IP range and reverse DNS: a 2026 reference
Verify AI bots by IP range and reverse DNS: the official IP lists and DNS checks for OpenAI, Anthropic, Google, Perplexity, Bing, Apple, Amazon and more.
AI training crawler
CCBot is the crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data. Others, including AI developers, can use that archive.
Common Crawl says to block CCBot by adding a User-agent: CCBot group with Disallow: /.
User-agent: CCBot Allow: /
Sill Agent Intelligence
Sill Agent Intelligence is building CCBot's behaviour profile from verified traffic across the Sill network: reach, identity checks, activity trend and the pages it targets. It appears here automatically when ready.
reverse DNS
Verify AI bots by IP range and reverse DNS: the official IP lists and DNS checks for OpenAI, Anthropic, Google, Perplexity, Bing, Apple, Amazon and more.
Block it if you do not want your pages in the Common Crawl archive. Because others reuse the archive, this is also how some sites keep content out of later datasets.
Common Crawl publishes IP ranges as JSON and says a forward and reverse DNS lookup should match a crawl.commoncrawl.org hostname. It warns that other crawlers sometimes falsely claim to be CCBot.