Skip to content
search-engine-optimizedby Netsolution

How robots.txt is interpreted

The robots.txt file controls crawling, not indexing: a blocked address can still appear in the results, without a snippet. The interpretation rules are published, including the arbitration between contradictory directives, the maximum file size and the handling of response codes.

A file about crawling, and about nothing else

The most widespread misunderstanding in the whole field comes down to a confusion between two verbs: crawl and index. The robots.txt file speaks only to the first.

The engine cannot index the content of a page it is disallowed from crawling, but it can still index the address and display it in the results without a snippet.

officialSource : Google Search Central, “How Google interprets the robots.txt specification”

The consequence is counter-intuitive and worth stating bluntly: for a page not to be indexed, it has to stay crawlable. The instruction asking for no indexing is read in the page or in its HTTP header, and a page disallowed from crawling is never read. Blocking and wanting to de-index are therefore two intentions that cancel each other out.

The welcome sign of an estate open to the public, listing the guide dogs allowed, a request not to disturb the peacocks, and the opening hours.
A sign at the gate, stating what visitors are asked to do and preventing nobody from anything. That is exactly this file's status: an instruction addressed to crawlers that agree to read it.Richard Webb — CC BY-SA 2.0 · Wikimedia Commons, relevé le

What the file recognises

Only four fields are supported: user-agent, allow, disallow and sitemap.

officialSource : Google Search Central, “How Google interprets the robots.txt specification”

Four fields. Everything else you see going past in files across the web — delay directives between requests, indexing instructions slipped into the file, fields invented by one tool or another — is not on that list. An unrecognised field is not a noisy error: it is ignored, silently, which is far more dangerous. You believe you have given an instruction.

The robots exclusion protocol is a standard published by the IETF, on the standards track, and not a convention belonging to one engine.

officialSource : IETF, RFC 9309, “Robots Exclusion Protocol”

That detail matters for a practical reason: the protocol is a published standard, readable by anyone, and one engine’s interpretation rules sit on top of that standard. The two documents are read together, and this is the only area of the field where the reference does not belong to the engine’s vendor.

Arbitration between contradictory rules

This is where intuition misleads most often, and it is documented precisely.

Arbitration is done on the length of the rule's path: the most specific rule applies, and in case of a conflict, wildcard rules included, the least restrictive one wins.

officialSource : Google Search Central, “How Google interprets the robots.txt specification”

Two mechanisms follow one another, and they have to be taken in order. Specificity first: it is the length of the rule’s path that decides, not its position in the file. A file is therefore not a sequence of instructions read from top to bottom, whatever its shape suggests. Direction next: at equal specificity, the least restrictive wins. A precise permission reopens what a broad disallow had closed, and that is how you keep reachable a file needed to render a page whose directory is otherwise disallowed.

Checking every rule against a list of real addresses before publishing it

hypothesis

Effort
low
Time to effect
immediate
Risk
none
Durability
structural

Two material limits nobody anticipates

A size limit is applied to the file, set at five hundred kibibytes, and anything past that limit is ignored.

officialSource : Google Search Central, “How Google interprets the robots.txt specification”

A size limit looks like a footnote until the day a site has piled up rules over the years, one per incident. What overflows is ignored — without warning, and therefore without anyone noticing that half the file no longer applies.

The file's content is generally cached for up to twenty-four hours, and sometimes longer when refreshing the cached version is impossible.

officialSource : Google Search Central, “How Google interprets the robots.txt specification”

Caching explains a phenomenon many people read as slowness on the engine’s part: a correction does not act straight away. It has a more unpleasant corollary — a block posted by mistake, then removed, keeps acting for a while. That is why a rule is checked before going live, and not by observing its effects.

Response codes, or the file you think you have

Crawlers treat every error in the 4xx series, apart from 429, as though no valid robots.txt file existed.

officialSource : Google Search Central, “How Google interprets the robots.txt specification”

In other words: a missing file amounts to a general permission. That is a reasonable design choice — the web would work badly if a missing file closed sites down — but the consequence has to be drawn. A robots.txt that answers with an error instead of its content protects nothing any more, and that failure is the kind that does not show on an otherwise perfectly working site.

Server errors follow a distinct, graduated logic: crawlers first pause, then lean on the cached version while retrying, and their long-term behaviour depends on the site’s general availability. The practical lesson is the same in both cases: the HTTP response of this file is part of what you monitor, on a par with that of the pages.

What does not work

To stop a page appearing in Google, just add it to robots.txt: it will disappear from the results.

The documentation says the opposite: the engine can index the address and display it without a snippet. The block prevents the page being read, and therefore prevents any no-index instruction inside it being read too. The advice produces the opposite of the result announced.

One last reflex deserves naming, because it looks prudent: blocking style and script files to “save” crawling. They are needed to render the page, and a page rendered without them is not the page visitors see. The saving is real; so is what it costs.

Sources

  1. How Google interprets the robots.txt specification — Google Search Centralaccessed on
  2. Introduction to robots.txt — Google Search Centralaccessed on
  3. RFC 9309 — Robots Exclusion Protocol — IETFaccessed on

Block revalidated on

Frequently asked questions

Does blocking a page in robots.txt remove it from the results?
No, and it is written in plain words: the engine can still index the address and display it without a snippet. The block prevents the content being read, not the page being mentioned. For a page not to be indexed, it has to stay crawlable instead.
What happens if the file returns an error?
An error in the 4xx series, apart from code 429, is treated as though no valid file existed, and therefore as a general permission. A missing file is not a closed site. Server errors follow a different logic, graduated over time according to how long the outage lasts.
Which rule wins when two rules contradict each other?
Arbitration is done on path length: the most specific rule applies. At equal specificity, wildcard rules included, the least restrictive one wins. A precise permission can therefore reopen a directory disallowed higher up in the file.
How long does a change take to be taken into account?
The file is cached, generally for up to twenty-four hours, and sometimes longer if refreshing is impossible. A correction is therefore not immediate, which has a practical consequence: a block posted by mistake keeps acting after being removed.