Content-Signal
Content-Signal in robots.txt: ai-train, ai-input and search explained
How the Content-Signal line in robots.txt works: ai-train, ai-input and search, Cloudflare's defaults, the IETF aipref drafts, and who honors them.
AI search crawler
PerplexityBot surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models.
Perplexity recommends allowing PerplexityBot in robots.txt so your site appears in search results; changes can take up to 24 hours.
User-agent: PerplexityBot Allow: /
Sill Agent Intelligence
Content-Signal
How the Content-Signal line in robots.txt works: ai-train, ai-input and search, Cloudflare's defaults, the IETF aipref drafts, and who honors them.
Block AI agents
Should you block AI bots and agents? A decision guide for store owners: training crawlers, AI search bots and shopping agents each need a different answer.
ExaSearchBot
What ExaSearchBot is, what it looks like in your logs, how to verify its Web Bot Auth signatures, and how to allow or block it with robots.txt.
Blocking it means your site may not appear in Perplexity search results. Allow it if you want Perplexity to link to you.
Perplexity says to match the user agent and check the source IP against its published JSON list, which it treats as the source of truth.