Glossary

What is robots.txt?

Definition

robots.txt is a plain text file at the top level of a website that lists which URLs automated crawlers may and may not fetch. It follows the Robots Exclusion Protocol, standardized as RFC 9309 in September 2022. It is a request that well-behaved crawlers honor. It does not block anyone and it does not control login or access.

Why it matters

robots.txt is where you tell crawlers, including AI companies' crawlers, what you want. OpenAI, for example, lets you allow its search crawler while disallowing its training crawler, so you can appear in its search results but ask that your content not be used for model training.

It is also easy to over-trust. It states your preference. It does not enforce it. For the practical setup, see robots.txt for AI agents.

How does robots.txt work?

The file lives at one fixed address. RFC 9309 says: "The rules MUST be accessible in a file named "/robots.txt" (all lowercase) in the top-level path of the service." For example, https://example.com/robots.txt.

Inside, you name a crawler by its user agent token and list what it may or may not fetch. When two rules conflict, the RFC says "The most specific match found MUST be used."

Crawlers should not read a stale copy for long. The RFC says they "SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable." OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust.

Server errors matter too. If your robots.txt returns a 4xx status, the RFC says a crawler "MAY access any resources on the server." If it returns a 5xx status, the crawler "MUST assume complete disallow."

What can robots.txt not do?

It cannot keep a visitor out. The RFC states: "These rules are not a form of access authorization." Google adds that robots.txt "is not a mechanism for keeping a web page out of Google" and points to a noindex directive or password protection instead.

It also does not reach every kind of bot. OpenAI says it uses robots.txt tags for its search crawler, OAI-SearchBot, and its training crawler, GPTBot, and that "Each setting is independent of the others." For ChatGPT-User, it says: "Because these actions are initiated by a user, robots.txt rules may not apply." That is the idea behind a user-triggered fetcher. So check each bot separately rather than assuming one rule covers them all.

Finally, a request that claims a bot's name can be false. To tell real crawlers from fakes, see verified bot.

  • llms.txt is a separate, proposed file that points AI tools to your most useful pages. It is not part of RFC 9309.
  • Content signals are a line inside robots.txt that states how you want your content used after it has been accessed.

Also called: Robots Exclusion Protocol, robots exclusion standard, REP

Sources

  1. RFC 9309: Robots Exclusion Protocol · checked 21 Nov 2026
  2. Google Search Central: Introduction to robots.txt · checked 21 Nov 2026
  3. OpenAI: Overview of OpenAI crawlers · checked 21 Nov 2026

The Sill Signals Team · Updated 21 Nov 2026

See which AI agents visit your site, and decide what they may do.

Scan your site free