ExaSearchBot
ExaSearchBot: the signed crawler behind Exa search
What ExaSearchBot is, what it looks like in your logs, how to verify its Web Bot Auth signatures, and how to allow or block it with robots.txt.
AI search crawler
ExaSearchBot is the crawler Exa uses to discover and index pages on the public web, so they can be found, retrieved and cited through Exa's search engine and API. Exa says every ExaSearchBot request is signed with HTTP Message Signatures under the Web Bot Auth scheme.
Exa says ExaSearchBot respects robots.txt. It looks for a group addressed to ExaSearchBot; without one it follows the rules you set for major search engines, otherwise the * group. Exa says robots.txt controls crawling, not indexing, so use noindex to remove pages already indexed.
User-agent: ExaSearchBot Allow: /
Sill Agent Intelligence
Sill Agent Intelligence is building ExaSearchBot's behaviour profile from verified traffic across the Sill network: reach, identity checks, activity trend and the pages it targets. It appears here automatically when ready.
ExaSearchBot
What ExaSearchBot is, what it looks like in your logs, how to verify its Web Bot Auth signatures, and how to allow or block it with robots.txt.
Only if you do not want your pages in Exa's index. Exa says the index lets users find, retrieve and cite your content and send people back to your site.
Exa's crawler page lists no IP ranges for it. Verify the request signature against Exa's public key directory instead; Exa says this holds regardless of the request's IP address.