reverse DNS
Verifying AI bots by IP range and reverse DNS: a 2026 reference
Verify AI bots by IP range and reverse DNS: the official IP lists and DNS checks for OpenAI, Anthropic, Google, Perplexity, Bing, Apple, Amazon and more.
Fetches pages for a user
Meta-ExternalFetcher fetches individual links at a user's request. Meta says it supports functions such as evaluating and improving agentic AI capabilities.
Meta says it may bypass robots.txt rules because it performs fetches requested by the user.
User-agent: meta-externalfetcher Allow: /
Meta does not publish a way to verify Meta-ExternalFetcher. Treat the user agent alone as a claim, not proof.
Sill Agent Intelligence
Sill Agent Intelligence is building Meta-ExternalFetcher's behaviour profile from verified traffic across the Sill network: reach, identity checks, activity trend and the pages it targets. It appears here automatically when ready.
reverse DNS
Verify AI bots by IP range and reverse DNS: the official IP lists and DNS checks for OpenAI, Anthropic, Google, Perplexity, Bing, Apple, Amazon and more.
Meta's AI crawlers
Meta's AI crawlers explained: what Meta-ExternalAgent, Meta-ExternalFetcher and Meta-WebIndexer do, how robots.txt applies, and which ones to allow.
Meta says a robots.txt rule may not be effective because the fetch was requested by a user. Firewall rules are the firmer option.
Meta's crawler page does not publish IP ranges or a verification method for it.