Robots Exclusion Protocol

The Robots Exclusion Protocol (REP) lets a website tell automated web agents which parts of the site they should or should not crawl. It communicates these preferences through a publicly accessible robots.txt file containing rules that cooperating crawlers read before requesting website resources.

How a website communicates its preferences

The protocol is standardised in RFC 9309. A website places a plain-text file named robots.txt at the root of its service, for example, https://example.org/robots.txt.

• An automated agent retrieves this file and identifies the rules applicable to it before crawling other resources.
• The file organises instructions into groups, each identifying one or more crawlers and then specifying the paths they may or may not crawl.
• These instructions describe crawling preferences rather than directly blocking requests at the server.

How an automated agent interprets the rules

The main fields are User-agent, which identifies the crawler addressed, and Allow and Disallow, which specify permitted and restricted paths.

• A User-agent value of * provides a fallback group for crawlers that have no specifically matching group.
• For example, a group containing User-agent: * and Disallow: /private/ asks such crawlers not to request paths beginning with /private/.
• An Allow rule can permit a narrower path within a broader disallowed area.
• When multiple path rules match a requested resource, the most specific match, meaning the longest matching path, takes precedence.
• If equivalent Allow and Disallow rules conflict, the Allow rule should be used.
• If no applicable path rule matches, access is allowed under the protocol; this does not override other access restrictions.

What the protocol does not establish

REP depends on crawler cooperation and is not an access-control mechanism. A non-cooperating agent can disregard the published instructions.

• Confidential resources require actual protection, such as authentication and authorisation; listing a path in robots.txt does not secure it.
• Because robots.txt is public, listing sensitive paths can reveal their locations.
• Crawling instructions do not themselves settle copyright permissions for copying, adapting or reusing website content.