Skip to content
search-engine-optimizedby Netsolution

Crawl budget

Crawl budget is what an engine devotes to downloading a site's pages. It comes out of two factors: what the server can take, and how much the engine wants to come back. The documentation explicitly reserves the subject for very large sites, or for those that change very fast.

Who exactly is this subject for?

Let us start with the most useful piece of information in this guide, and the one most often left out of articles on the topic: the documentation describing crawl budget states itself who it is written for.

The documentation reserves this subject for three profiles: very large sites whose content changes moderately often, medium-sized sites whose content changes very fast, and those where a large share of addresses is classed “discovered, currently not indexed”.

officialSource : Google Search Central, “Optimize your crawl budget”

The orders of magnitude cited are a million unique pages for the first profile, ten thousand for the second. Which is to say that most sites reading advice about crawl budget fall into none of the three categories. And the source drives the point home:

If your pages appear indexed on the day they are published, that guide does not concern you.

officialSource : Google Search Central, “Optimize your crawl budget”

That is a one-criterion test, applicable in a minute. If your pages enter the index on the day you publish them, the rest of this guide is general knowledge rather than a job on your list.

What is crawl budget made of?

Two distinct things, which the documentation names separately. The first is a technical constraint:

The crawl capacity limit bounds the total time the server spends holding connections open for the engine, taking into account the number of simultaneous connections and how long they last.

officialSource : Google Search Central, “Optimize your crawl budget”

It describes a capacity in the sense a systems administrator means it: how many simultaneous connections, for how long. A slow or unstable server reduces that capacity, not as a punishment but as a precaution — a crawler that brought a site down would be doing nobody a favour.

The second is a constraint of interest:

Crawl demand depends among other things on the size of the site, how often it is updated, the quality of its pages and their relevance, compared with those of other sites.

officialSource : Google Search Central, “Optimize your crawl budget”

That sentence is worth reading slowly, because it contains the two words technical optimisation cannot reach — “quality” and “relevance” — and above all the closing comparison, “compared with those of other sites”. In other words, the appetite for coming back is not an internal property of the site: it is relative.

The cover of an American ration book from 1943, carrying a printed number, identity boxes to fill in, and the words “Not valid without stamp”.
An allowance fixed in advance, spent coupon by coupon and never argued with. Crawling behaves the same way: every address served consumes part of it, including the ones that did not deserve serving.Bill Faulk — Domaine public · Wikimedia Commons, relevé le

Where does the budget actually go, on the sites concerned?

On the sites concerned, most of the wasted crawling comes from addresses produced automatically by filters, sorts and pagination, rather than from written pages.

hypothesis

That “hypothesis” status marks an observation from practice, not a published measurement. It has a direct consequence: on a faceted site, the number of reachable addresses grows as the product of the options available, while the number of genuinely distinct pages barely grows at all. Three combinable filters with five values each are enough to manufacture more addresses than the catalogue has products.

And the crawler has no way of knowing in advance which of them are useful:

A crawler does not download every page it discovers: some are disallowed by the site, others require authentication.

officialSource : Google Search Central, “In-depth guide to how Google Search works”

Reducing the useless addresses produced by filters and sorts

hypothesis

Effort
high
Time to effect
months
Risk
low
Durability
structural

The risk axis is not at its lowest, and that matters. Removing addresses from the crawl queue is an operation that easily hits the wrong target: a parameter judged useless may carry a page people genuinely ask for, and a block then prevents any correction by the page itself. This work is done from an inventory, never as a precaution.

How do you decide, rather than apply recipes?

Three questions, in this order.

First: am I in one of the three documented profiles? If not, the matter stops there, and the time is better spent elsewhere. Then: what share of my addresses stays lastingly at the discovered-but- not-crawled stage? That is the signal the source names, and it is read in the indexing report rather than in a third-party tool. Finally, and only then: which families of addresses have no reason to exist, and by which mechanism do we make them go away — delete, group under a canonical, or block.

The order matters. Each of those three interventions carries a development cost and a risk of error; running them without having answered the first two questions amounts to paying off a debt you never took on.

What does not work, or no longer works

Improve your crawl budget by deleting useless pages, cutting click depth to three, and submitting a sitemap every day to flag your new content.

These gestures are presented as a universal setting when the documentation restricts the subject to three site profiles. The numbered depth is published nowhere, and submitting a sitemap daily changes neither the server's capacity nor the engine's interest.

One more stubborn confusion is worth naming: crawling and ranking are two different stages. A site crawled more is not a site placed better; it is a site whose pages are examined more often. Conflating the two turns infrastructure work into a promise of position, which nothing in the documentation supports.

Sources

  1. Optimize your crawl budget — Google Search Centralaccessed on
  2. In-depth guide to how Google Search works — Google Search Centralaccessed on
  3. Introduction to robots.txt — Google Search Centralaccessed on
  4. Page indexing report — Google Search Consoleaccessed on

Block revalidated on

Frequently asked questions

Does my hundred-page site have a crawl budget problem?
Almost certainly not. The official documentation explicitly rules the subject out for sites whose pages appear indexed on the day they are published. Trying to optimise a constraint that does not apply amounts to treating a symptom you do not have.
Does blocking pages in the exclusion file improve crawling?
It removes addresses from the queue, which helps when they are massively useless. But a block also stops any instruction carried by those pages being read, a no-index instruction included. The gesture is thought through, not applied as a precaution.
Does a faster site get crawled more?
The capacity limit depends on the time the server spends holding connections open, so a server that responds better can support more of them. That does not increase the engine's appetite for coming back, which depends on other factors entirely.
How do I know whether crawling really is my problem?
By looking at the share of your addresses classed “discovered, currently not indexed” in the indexing report. A large and lasting share is the signal the documentation names. A handful of pages in that state is not a diagnosis.