Skip to content
search-engine-optimizedby Netsolution

Verifying that a crawler is Googlebot

A user agent name is declared, so it can be invented. The published method is to resolve the address into a name, check that the name ends with an expected domain, then resolve that name back into an address and find the same one. Published address lists allow the same check at scale.

What a log line does not tell you

In a server’s logs, a crawler visit appears as a user agent name: a string of characters the client sends with its request. It reads easily, and that is all that can be said for it.

A user agent name is a string the client chooses to send: it proves nothing by itself, which the very existence of a published verification procedure indirectly confirms.

hypothesis

The status is that of a reading, and the reasoning fits in a sentence: the documentation publishes a verification procedure, therefore the declared name is not enough. It even gives the motive for that page — the case of a site worried about visitors claiming to be from Google.

The method, in three steps

The manual verification described comes down to three steps: run a reverse DNS lookup on the accessing address from the logs, verify that the domain name obtained is one of the three expected domains, then run a forward lookup on that name and find the original address.

officialSource : Google, “Verify requests from Google crawlers and fetchers”

Why do both lookups rather than one is worth understanding, because it decides what the check is worth. The reverse lookup of an address is filled in by whoever holds that address: it can announce whatever it likes. The forward lookup of the resulting name depends on the domain named instead. Doing both and finding the same address therefore requires controlling both sides.

The name obtained must belong to one of the three domains the documentation names, and to those alone: googlebot.com, google.com or googleusercontent.com.

officialSource : Google, “Verify requests from Google crawlers and fetchers”

Three domains, and no others. A name containing “google” without ending in one of the three establishes nothing, and that is exactly the shape the attempts found in logs take.

The example given in the documentation still checks out: the address 66.249.66.1 resolves to crawl-66-249-66-1.googlebot.com, and that name resolves back to the same address.

observedSource : Lookup run from a machine outside the vendor's networks, 2026-08-25

The lookup was rerun for this page, on the documentation’s own example, and it agrees. The status stays observed rather than tested: two resolution commands do not amount to a protocol, and this site reserves the second status for measurements described precisely enough to be replayed identically.

Our firewall identifies Googlebot by its user agent and reserves priority bandwidth for it, to improve your crawl budget.

A user agent is copied in one line. A rule granting favourable treatment on that criterion alone grants it first to those who bother to lie, which is precisely what the published verification procedure exists to prevent.

Three families, three behaviours

Three families of crawlers are distinguished, with different behaviours: common crawlers, which always respect robots.txt rules for automatic crawls; special-case crawlers, which may or may not respect them; and user-triggered fetchers, which ignore them.

officialSource : Google, “Verify requests from Google crawlers and fetchers”

That distinction explains an observation that comes up often and usually passes for an anomaly: an address disallowed in robots.txt is fetched anyway. If the request comes from a user-triggered fetcher, the documentation says it ignores those rules, and the reason is coherent — the request comes from a user, not from a crawling program.

The suffixes differ by family too, which makes the check more legible: a name under googlebot.com denotes a common crawler, a name under google.com may denote a special-case crawler, and a user-triggered fetcher may resolve to a hosting name.

Checking at scale

For large scale checking, the documentation points to published address lists in JSON, one per family of crawlers, to be matched against the address accessing the site.

officialSource : Google, “Verify requests from Google crawlers and fetchers”

Querying DNS for every line of a log does not hold up: the volume is too large and the latency adds up. The published lists allow a local comparison, range by range, with no network request at all. They carry a trade-off worth naming: they change, and a copy frozen inside a firewall ages without warning.

Verifying where crawlers come from before applying a filtering rule

hypothesis

Effort
moderate
Time to effect
immediate
Risk
none
Durability
structural

The effort comes down to two commands for a one-off check, and a little code for a regular one. The delay is immediate, since the answer arrives with the lookup. The risk is nil for the verification itself — it sits entirely on the side of the filtering it precedes. Durability is good: the method has been published in the same form for years, and only the address lists change.

What does not work, or no longer works

Trusting the user agent alone, first. It is the client’s own declaration, and it is worth what a stranger’s word is worth.

Then pasting a once-collected address list into a firewall. The published lists exist to be read again, and a range removed from the list will still be allowed by the copy.

Finally, concluding from a name containing “google” that the request comes from Google. Three domains are named, the check applies to the end of the name, and the second lookup remains indispensable.

Sources

  1. Verify requests from Google crawlers and fetchers — Googleaccessed on
  2. How HTTP status codes, and network and DNS errors, affect Google Search — Google Search Centralaccessed on

Block revalidated on

Frequently asked questions

Why not trust the user agent name?
Because it is declared by whoever requests the page, and a string of characters can be copied. The very existence of a published verification procedure says as much: if the name were enough, the procedure would have no reason to exist. The logs of any visited site contain fake Googlebots.
Is a single DNS lookup enough?
No, and that is the point the published method insists on describing in two steps. The reverse lookup gives a name, which the holder of the address partly controls. The forward lookup of that name must then return the original address. It is the agreement of the two that establishes anything.
What should be done with a crawler that fails the test?
Treat it like any anonymous client: the check establishes only that it is not what it claims to be. Blocking, rate limiting or ignoring is a matter of site policy. Take care not to confuse the three families, two of which carry suffixes other than googlebot.com.
Do all of Google's crawlers respect robots.txt?
No, and the documentation says so plainly. Only common crawlers always respect the rules during automatic crawls. Special-case crawlers may or may not respect them, and user-triggered fetchers ignore them, because the request comes from a person rather than from a crawling program.
How do you check thousands of log lines?
Through the published address lists in JSON, one per family, which the documentation provides in CIDR notation. The comparison then happens without querying DNS for every line. It is the recommended method as soon as the volume goes past a few one-off checks.