AI agent directory
Who runs each AI crawler, assistant and search bot, what it does on your website, and how to check that a request really came from it.
Amazon
- AmazonbotAI training crawler
- Amzn-SearchBotAI search crawler
- Amzn-UserFetches pages for a user
Anchor Browser
- Anchor BrowserFetches pages for a user
Anthropic
- Claude-AgentFetches pages for a user
- Claude-SearchBotAI search crawler
- Claude-UserFetches pages for a user
- ClaudeBotAI training crawler
Apple
- ApplebotSearch engine crawler
- Applebot-Extendedrobots.txt control token
Browserbase
- BrowserbaseFetches pages for a user
Common Crawl
- CCBotAI training crawler
DuckDuckGo
- DuckAssistBotFetches pages for a user
Exa
- ExaSearchBotAI search crawler
- Google-AgentFetches pages for a user
- Google-Extendedrobots.txt control token
- Google-GeminiNotebookFetches pages for a user
- GooglebotSearch engine crawler
Meta
- FacebookExternalHitLink preview fetcher
- Meta-ExternalAgentAI training crawler
- Meta-ExternalFetcherFetches pages for a user
- Meta-WebIndexerAI search crawler
Microsoft
- BingbotSearch engine crawler
Mistral
- MistralAI-UserFetches pages for a user
Moonshot AI
- KimiBotAI training crawler
OpenAI
- ChatGPT agentFetches pages for a user
- ChatGPT-UserFetches pages for a user
- GPTBotAI training crawler
- OAI-AdsBotAds review crawler
- OAI-SearchBotAI search crawler
Perplexity
- Perplexity-UserFetches pages for a user
- PerplexityBotAI search crawler