Skip to content
search-engine-optimizedby Netsolution

AI crawlers and Google-Extended

Google-Extended is a robots.txt token with no user agent of its own: it decides whether content may be used to train Gemini models and to ground them at prompt time. It has no effect on inclusion in Search, and therefore none on the AI answers that are part of Search.

A token that crawls nothing

The first thing to understand about Google-Extended is that it does not name a crawler. It is a label you write in a file, and nothing on the network side corresponds to it.

Google-Extended has no separate HTTP user agent string: crawling is done with existing Google user agents, and the token exists only in robots.txt, in a control capacity.

officialSource : Google, “Google's common crawlers”

The immediate consequence saves hours: there is no point looking for that name in a server’s logs. Crawling happens with the usual agents; the token only declares a preference about what is done with the result afterwards.

The token lets publishers manage whether crawled content may be used to train future generations of Gemini models and for grounding, meaning providing content to the model at prompt time to improve factuality and relevancy.

officialSource : Google, “Google's common crawlers”

Two uses, then, and they are worth separating. Training concerns models to come. Grounding concerns the moment of the question: supplying content to the model to improve the accuracy of its answer. That second use is the one most publishers actually have in mind when they talk about being cited.

The misunderstanding that costs most

Google-Extended does not impact a site's inclusion in Google Search, nor is it used as a ranking signal.

officialSource : Google, “Google's common crawlers”

The sentence is reassuring and it is regularly read backwards. It says that blocking the token costs nothing in Search; it does not say that blocking it removes you from AI answers displayed in Search. The other document settles that point.

Since AI is built into Search, robots.txt directives for Googlebot are the control over access for those features; to limit what is shown from pages, the vendor points to the nosnippet, data-nosnippet, max-snippet and noindex rules.

officialSource : Google Search Central, “AI features and your website”

In other words, the two mechanisms do not overlap. What is built into Search is controlled with the means of Search: access through directives addressed to Googlebot, display and reuse through the snippet rules. The token governs separate products.

One line in your robots.txt and your content will no longer feed Google's AI-generated answers, with no impact on your rankings.

The second half is accurate, the first is not. The token quoted covers training and grounding of models in other products; the answers displayed in Search fall under directives addressed to Googlebot and the snippet rules.

The other tokens, and what they cover

GoogleOther is a generic crawler that various product teams may use to fetch publicly accessible content, for example for one-off research and development crawls; preferences addressed to it affect no specific product.

officialSource : Google, “Google's common crawlers”

Preferences addressed to Google-CloudVertexBot affect crawls requested by site owners themselves for building agents, and have no effect on Search or other products.

officialSource : Google, “Google's common crawlers”

The second case is worth flagging, because it is counter-intuitive: that token covers crawls site owners request themselves, in order to build their own agents. Blocking it as a precaution means refusing yourself a service you might have wanted.

A token belongs to the vendor that publishes it and settles only that vendor's case: blocking Google-Extended says nothing to the other systems crawling the web, and the real scope of a decision is measured by how many tokens would have to be written.

hypothesis

The status is that of a reading, and it follows from how the mechanism is built: a token belongs to the vendor that publishes it and binds nobody else. Nothing in the documentation claims otherwise; we conclude that a matter of principle about AI is not implemented by one line, but by a list somebody has to maintain.

Deciding how AI systems may use your content, token by token

hypothesis

Effort
moderate
Time to effect
immediate
Risk
none
Durability
short-lived

The effort is low to write, real to maintain. The delay is immediate, a robots.txt being reread often. The risk is low for this particular token, the documentation stating that it affects neither presence in Search nor ranking. Durability is the lowest on this site: these names change, and a frozen list stops saying what it thinks it says.

What does not work, or no longer works

Hunting for Google-Extended in your logs. The token has no separate user agent, so it leaves no trace there.

Then blocking that token expecting to leave the AI answers in Search. Those features are part of Search and are controlled with the means of Search.

Finally, copying a list of crawlers to block out of an article. The names change, and a dated file gives the illusion of a decision still being applied.

Sources

  1. Google's common crawlers — Googleaccessed on
  2. AI features and your website — Google Search Centralaccessed on

Block revalidated on

Frequently asked questions

Does blocking Google-Extended remove you from AI answers in Search?
No. The token covers training Gemini models and grounding them in the corresponding apps. The overviews and the conversational mode are part of Search: robots.txt directives for Googlebot control access there, and the snippet rules limit what is shown of the page.
Does blocking that token hurt rankings?
The documentation answers directly: Google-Extended does not impact a site's inclusion in Search, nor is it used as a ranking signal. It is one of the few statements in this corpus phrased that plainly, and it can be quoted as it stands.
How do you spot Google-Extended in your logs?
You don't: it has no separate user agent, crawling being done with the existing agents. The token exists only in robots.txt, where it acts as a control. Hunting for its trace in logs is time spent looking for something that leaves none.
What is GoogleOther doing in this list?
It is a generic crawler used by various teams to fetch publicly accessible content, including one-off research and development crawls. Preferences addressed to it affect no particular product, which makes it a different case from the other two tokens.
Should every AI crawler be blocked?
That is an editorial decision rather than a technical one, and it carries a maintenance cost: each system publishes its own token, and a list written once ages. The useful question is not how to block, but what the site gains and loses by no longer being read by those systems.