AI search crawler

Meta-WebIndexer

Meta-WebIndexer is a Meta crawler that navigates the web to improve Meta AI search result quality for users. Meta says allowing it in robots.txt helps Meta cite and link to your content in Meta AI's responses.

Sources: Meta's own documentation and the Sill agent registry · checked 17 Nov 2026

Company
Meta
Acts for
Meta, for Meta AI search results
Follows robots.txt
Yes
How to verify
No method published

robots.txt

Meta instructs site owners to add a robots.txt disallow for the relevant crawler to block it, with no caveat for this one. Meta says allowing it helps Meta cite and link to your content in Meta AI.

User-agent: meta-webindexer
Allow: /

How to check it is really Meta-WebIndexer

Meta does not publish a way to verify Meta-WebIndexer. Treat the user agent alone as a claim, not proof.

How verification works →

Sill Agent Intelligence

What it does on websites PROFILE IN PROGRESS

Sill Agent Intelligence is building Meta-WebIndexer's behaviour profile from verified traffic across the Sill network: reach, identity checks, activity trend and the pages it targets. It appears here automatically when ready.

Other Meta agents

Read more on Signals

Questions about Meta-WebIndexer

Should I block Meta-WebIndexer?

Only if you do not want Meta AI to cite and link to your pages. Meta says allowing it helps Meta cite and link to your content in Meta AI's responses.

Can I verify it by IP address?

Meta's crawler page lists its user agent strings but does not publish IP ranges or a verification method for it.

Sources

  1. Meta: Meta Web Crawlers · checked 17 Nov 2026

Official documentation: https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers · Spotted an error? Tell us

See which AI agents visit your site, and decide what they may do.

Scan your site free