Short answer
ExaSearchBot is the crawler Exa uses to discover and index public web pages for Exa, a search engine and search API used by people and AI applications. Exa says every ExaSearchBot request is signed with HTTP Message Signatures under the Web Bot Auth scheme, so you can check it is genuine regardless of IP address. It follows robots.txt, and you can address it by the product token ExaSearchBot.
Key takeaways
- ExaSearchBot indexes public pages for Exa's search engine and API.
- Exa says every request is signed, so verify the signature.
- It follows robots.txt; use noindex to remove pages already indexed.
What is ExaSearchBot?
ExaSearchBot is the crawler that Exa uses to find and index pages on the public web. Exa's crawler page puts it simply: "Exa is a search engine. Similar to other search engines, Exa uses a robot (ExaSearchBot) to discover and index pages on the public web."
Exa also offers its search as an API for developers. Its home page describes it as a "Search API for AI agents that need real-time web data", and its documentation says Exa Search "takes a natural language query and returns ranked web results with clean page content." So when ExaSearchBot indexes your page, the people finding it may be using an AI application built on Exa, not only a search box.
Exa describes the purpose as "search and retrieval: connecting people and applications to your content and sending them back to your site."
What does ExaSearchBot look like in my logs?
It sends a short, clearly labelled user agent string. Exa publishes it as:
Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)
The robots.txt product token is ExaSearchBot. The link in the string goes to Exa's crawler page, which explains the bot and how to control it.
Exa also names two related user agents on the same page:
- ExaUser. Exa says pages it fetches "on a user's request, rather than by crawling, go out as ExaUser."
- ExaSearchBot-Images. Exa crawls images under this name.
If you search your raw server or CDN logs, look for all three names. Each one tells you something different: ExaSearchBot is crawling, ExaUser is fetching because someone asked, and ExaSearchBot-Images is collecting images. For the general difference between crawlers and on-demand fetches, see our glossary entry on the user-triggered fetcher.
Exa adds a warning worth taking seriously: "User-agent strings can be spoofed, so don't trust the string alone." A request that says ExaSearchBot only claims to be ExaSearchBot until you check it.
How do I verify that a request is really from ExaSearchBot?
Check the request signature. Exa says: "Every ExaSearchBot request is cryptographically signed using HTTP Message Signatures (RFC 9421), following the Web Bot Auth scheme."
Each request carries three headers: Signature, Signature-Input and a Signature-Agent header that Exa says identifies Exa. To check a request yourself, Exa tells you to fetch its public key directory and validate the signature against it. The directory lives at:
https://crawler.exa.ai/.well-known/http-message-signatures-directory
Exa says the keys there are Ed25519 keys. When we fetched the directory, it returned a short JSON list of keys.
This is the main thing that makes ExaSearchBot different from many crawlers. Exa does not ask you to match IP ranges. It says: "Because verification is cryptographic, it holds regardless of the request's IP address." The upshot:
- A request with a valid Exa signature is from Exa.
- A request that says ExaSearchBot but has no signature, or a signature that fails, is a request that claims to be ExaSearchBot but fails verification. Treat it as an unknown bot.
Exa also says that on Cloudflare, Akamai and other providers that support Web Bot Auth, ExaSearchBot "is validated automatically at the edge", with nothing for you to configure. That is Exa's statement about how those providers handle its signatures. Cloudflare's own documentation says a bot it marks as Verified must declare who it is through a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS, and must obey robots.txt and crawl directives.
For how these signatures work in general, see Web Bot Auth explained. For the wider problem of bots that borrow well-known names, see fake AI crawlers.
Does ExaSearchBot follow robots.txt?
Yes. Exa says: "ExaSearchBot respects the Robots Exclusion Protocol."
The way it picks which rules to follow is worth knowing, because it may follow rules you didn't write for it. Exa says it looks for a group addressed to ExaSearchBot. "If you haven't provided one, it follows the rules you've set for major search engines; otherwise it follows the * group." Exa's page does not list which search engines it means.
So if you already restrict search engine crawlers, ExaSearchBot may be following those rules today. To set its rules directly, give it its own group, which takes precedence:
User-agent: ExaSearchBot
Disallow: /private/
To block it from your whole site:
User-agent: ExaSearchBot
Disallow: /
Robots.txt stops crawling, not indexing. Exa says: "robots.txt controls crawling, not indexing." To remove a page that is already in Exa's index, use a noindex robots meta tag or an X-Robots-Tag: noindex header. Exa says ExaSearchBot drops the page after it next re-fetches it. That means the crawler has to be able to fetch the page again to see the tag.
For a wider view of which bots to name in robots.txt, see robots.txt for AI agents.
How does ExaSearchBot behave on my site?
Exa lists four behaviours on its crawler page:
| What Exa says | What it means for you |
|---|---|
| Respects robots.txt, "checked before fetch" | Your rules are read before each fetch, using the precedence above. |
| Signs requests "so you can confirm a request is genuinely us" | You can verify it without an IP list. |
| Crawls politely, "rate-limited per site" | Exa says this is to avoid putting load on your servers. |
| Public content only | Exa says it "does not attempt to bypass logins, paywalls, or CAPTCHAs, and submits no forms." |
These are Exa's own statements. If you see behaviour that doesn't match, first check the signature. A request that fails verification isn't Exa's.
Should I allow or block ExaSearchBot?
For most stores and business sites, allowing it makes sense. ExaSearchBot is a search crawler, and Exa says its index lets users "find, retrieve, and cite your content." If AI applications built on Exa are going to answer questions about products like yours, being in the index gives your pages a chance to be the cited source.
Reasons you might limit it:
- Private or thin pages. Use a path rule, such as the
/private/example above, rather than a full block. - Content you don't want reused in search results. Block it in robots.txt, and use
noindexfor pages already indexed. - Load. Exa says it rate-limits per site. If you still see heavy traffic under the ExaSearchBot name, check the signatures before blaming Exa.
Whatever you choose, write the rule against the ExaSearchBot token, and decide separately about ExaUser, which Exa says fetches pages on a user's request. Our guide to Cloudflare's signed agents and verified bots explains how signed crawlers fit into Cloudflare's bot settings if your site sits behind Cloudflare.
How Sill helps. Sill shows which AI agents and bots visit your site and the pages they read. It verifies signed agent requests (Web Bot Auth) on Cloudflare and WordPress installs, so a signed crawler like ExaSearchBot can be told apart from requests that only borrow its name. To see how ready your site is for agents, run the free scan.
Questions people ask
Does Exa publish IP ranges for ExaSearchBot?
Exa's crawler page does not list IP ranges. It points you to its public key directory instead and says that because verification is cryptographic, it holds regardless of the request's IP address.
Is ExaSearchBot the same as Exabot?
Exa's crawler page names three user agents: ExaSearchBot, ExaUser and ExaSearchBot-Images. It does not mention Exabot, so don't assume robots.txt rules written for Exabot apply to ExaSearchBot. Use the ExaSearchBot token.
Does ExaSearchBot fill in forms or log in to my site?
Exa says it fetches public content only. It does not attempt to bypass logins, paywalls or CAPTCHAs, and it submits no forms.
How do I contact Exa about its crawler?
Exa's crawler page gives support@exa.ai and asks you to write from an email address on the domain you are contacting it about, which helps Exa verify you.
Sources
- Exa: Exa Search Crawler (last updated 13 November 2026) · checked 17 Nov 2026
- Exa: ExaSearchBot public key directory · checked 17 Nov 2026
- Exa: Search API for AI agents (home page) · checked 17 Nov 2026
- Exa: Exa Search API documentation · checked 17 Nov 2026
- Cloudflare: Verified bots (last updated 12 November 2026) · checked 17 Nov 2026