Meta's AI crawlers
Meta's AI crawlers: Meta-ExternalAgent, Meta-ExternalFetcher and Meta-WebIndexer
Meta's AI crawlers explained: what Meta-ExternalAgent, Meta-ExternalFetcher and Meta-WebIndexer do, how robots.txt applies, and which ones to allow.
Link preview fetcher
FacebookExternalHit crawls a website or app when its link is shared on Meta's apps, such as Facebook, Instagram or Messenger. Meta says it gathers, caches and displays the title, description and thumbnail image for the link.
Meta says to block its crawlers with a robots.txt disallow for the relevant crawler, but that FacebookExternalHit might bypass robots.txt when performing security or integrity checks, such as checking for malware.
User-agent: facebookexternalhit Allow: /
Meta does not publish a way to verify FacebookExternalHit. Treat the user agent alone as a claim, not proof.
Sill Agent Intelligence
Meta's AI crawlers
Meta's AI crawlers explained: what Meta-ExternalAgent, Meta-ExternalFetcher and Meta-WebIndexer do, how robots.txt applies, and which ones to allow.
Usually not. Meta uses it to show the title, description and thumbnail when someone shares your link, so blocking it can leave shared links without a preview.
Meta's crawler page says allow-listing the crawler's IP addresses is more secure than allow-listing its user agent strings, but the page does not list those addresses or describe another verification method.