Agent spotlights

Meta's AI crawlers: Meta-ExternalAgent, Meta-ExternalFetcher and Meta-WebIndexer

Listen · 8 min

Narrated by an AI voice. Code examples are on the page.

Short answer

Meta documents three crawlers tied to its AI work. Meta-ExternalAgent crawls for uses such as training foundation AI models, Meta-WebIndexer improves Meta AI search results and helps Meta cite and link to your content, and Meta-ExternalFetcher fetches single links when a user asks. Each has its own robots.txt token, but Meta says Meta-ExternalFetcher may bypass robots.txt.

Key takeaways

  • Meta-ExternalAgent is for AI training and indexing; block it separately.
  • Meta-WebIndexer helps Meta AI cite and link to your pages.
  • Meta-ExternalFetcher acts for a user and may bypass robots.txt.

What are Meta's AI crawlers?

Meta's AI crawlers are three bots that Meta lists on its developer page for web crawlers: Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher. Meta describes uses for them that include training foundation AI models, improving Meta AI search results, and fetching links when a user asks.

The same page lists two more Meta crawlers that are not described in AI terms: FacebookExternalHit, which builds link previews, and Meta-ExternalAds, which Meta says is used for "improving advertising and other business-related products and services."

Meta's page says it lists "the User Agent (UA) strings that identify Meta's most common web crawlers and what each of those crawlers are used for." At the time of writing, the page shows an update date of 21 May 2026. It now lives under developers.facebook.com/documentation/, so update any bookmark that points to an older address.

Meta also says it prefers standard controls: it supports "industry-standard practices like robots.txt rather than non-standard formats like NoAI tags."

Which Meta crawler is which?

Each crawler has its own user agent token and its own job. This table uses Meta's own descriptions.

Crawler User agent token What Meta says it does robots.txt
Meta-ExternalAgent meta-externalagent Crawls for "use cases such as training foundation AI models or improving products by indexing content directly." Block with a disallow rule.
Meta-WebIndexer meta-webindexer Crawls "to improve Meta AI search result quality for users." Meta says allowing it helps Meta cite and link to your content.
Meta-ExternalFetcher meta-externalfetcher "fetches individual links at a user's request" Meta says it "may bypass robots.txt rules."
FacebookExternalHit facebookexternalhit Fetches pages shared on Facebook, Instagram or Messenger to show a title, description and thumbnail. May bypass robots.txt for security or integrity checks.
Meta-ExternalAds meta-externalads Crawls for advertising and other business-related products and services. Block with a disallow rule.

Meta says the user agent you see in your logs "will be similar to" strings such as:

meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers)
meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)
meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)

Shorter forms without the bracketed link, such as meta-externalagent/1.1, are listed too.

What is the difference between Meta-ExternalAgent and Meta-WebIndexer?

Meta-ExternalAgent is the crawler Meta links to AI training. Meta-WebIndexer is the one Meta links to Meta AI's search answers and citations.

Meta describes Meta-WebIndexer like this: "Meta analyzes online content to enhance the relevance and accuracy of Meta AI. Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses."

That split matters for a store. If you do not want your pages used for training foundation models, Meta-ExternalAgent is the crawler Meta associates with that. If you want Meta AI to be able to cite and link to your product pages, Meta says allowing Meta-WebIndexer helps. Note that Meta's description of Meta-ExternalAgent also mentions "improving products by indexing content directly", so it is not described as training-only.

This mirrors how other companies separate their bots. Our guide to whether to block AI agents covers the general difference between training crawlers, search crawlers and agents.

Does Meta respect robots.txt?

For most of its crawlers, Meta tells you to use robots.txt: "In order to block these crawlers, add a disallow for the relevant crawler to robots.txt." There are two documented exceptions.

  • Meta-ExternalFetcher: "The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user." Meta says it supports product functions including "helping AI navigate websites to complete tasks for users." That makes it a user-triggered fetcher: it loads a page because a person asked, not as part of a crawl.
  • FacebookExternalHit: Meta says it "might bypass robots.txt when performing security or integrity checks, such as checking for malware or malicious content."

Changes are not instant. Meta says: "Please allow up to 24 hours for changes to robots.txt to take effect because crawlers may cache the contents of robots.txt for up to 24 hours."

How can I tell if a request really comes from Meta?

On Meta's current page, the documented signal is the user agent string. Meta does not publish an IP list, reverse DNS hostname or request signature there for any of these crawlers.

The page does say, in the requirements for FacebookExternalHit, to add to your allow list "either the user agent strings or the IP addresses (more secure) used by the crawler." But it does not list those IP addresses.

A user agent is only a label, and anyone can send meta-externalagent/1.1. So treat a match as a claim, not proof. If a request carrying a Meta token behaves oddly, such as hammering login pages or ignoring a disallow for a crawler that Meta tells you to block that way, it may be a request that claims to be Meta but is not. Our guides to fake AI crawlers and IP ranges and reverse DNS explain how to check visitors where vendors do publish more.

Which Meta crawlers should an online store allow?

It depends on what you want from Meta. Here is one possible setup for a store that wants Meta AI to be able to cite it but does not want Meta-ExternalAgent crawling. Adjust it to your own choice.

User-agent: meta-webindexer
Allow: /

User-agent: meta-externalagent
Disallow: /

User-agent: facebookexternalhit
Allow: /

A few points to weigh:

  1. Keep FacebookExternalHit allowed if people share your product links. Meta says it shows the title, description and thumbnail for shared links.
  2. Decide on Meta-ExternalAgent on its own merits. Blocking it is the documented way to opt out of what Meta describes as training and direct indexing.
  3. Allow Meta-WebIndexer if Meta AI citations matter to you. Meta says allowing it helps Meta cite and link to your content.
  4. Do not count on robots.txt for Meta-ExternalFetcher. If you need to stop it, a firewall rule on its user agent is the firmer option, with the caveat that it would also turn away visits a real Meta AI user asked for.

If you sell on Shopify, Meta is also a sales channel. Shopify's help center describes a Meta channel where customers can buy through a Shopify-powered checkout on Meta surfaces. See our Shopify Agentic Storefronts guide. For robots.txt rules covering other companies' bots, see robots.txt for AI agents.

How Sill helps. Sill shows you which AI agents and bots visit your site, including Meta's crawlers, and the pages they read, so you can see the effect of your robots.txt choices. You can check how your site looks to agents with the free readiness scan.

Questions people ask

Does blocking Meta-ExternalAgent stop my links previewing on Facebook?

Meta documents them as separate crawlers with separate user agent tokens. Link previews come from FacebookExternalHit, so a robots.txt rule for meta-externalagent names a different crawler.

How long does Meta take to notice a robots.txt change?

Meta says to allow up to 24 hours, because its crawlers may cache the contents of robots.txt for up to 24 hours.

Does Meta publish IP ranges for its AI crawlers?

Not on its current crawler page. The page says allow-listing the crawler's IP addresses is more secure than allow-listing user agent strings, but it does not list the addresses or describe another way to verify a request.

Who do I contact at Meta about its crawlers?

Meta's crawler page gives webmasters@meta.com as the contact for questions or concerns.

Sources

  1. Meta: Meta Web Crawlers · checked 17 Nov 2026
  2. Shopify Help Center: Agentic storefronts, Meta · checked 17 Nov 2026

Written by The Sill Signals Team. Last reviewed 17 Nov 2026.

Share this articleLinkedInXBlueskyEmail

Keep reading

See which AI agents visit your site, and decide what they may do.

Scan your site free