How a search engine works
A search engine works in three separate stages: it crawls the web to download pages, it analyses and then stores what it downloaded, and it finally decides what to show for a given query. Each stage can fail independently of the other two, and none of them is guaranteed.
What are the three stages, and why separate them?
The temptation, when you first meet the subject, is to treat “being in Google” as a single state: you are in or you are out. That picture produces more wrong diagnoses than any other, because it hides where the problem actually sits.
A search engine's documentation describes three separate stages: crawling, indexing, then serving results.
officialSource : Google Search Central, “In-depth guide to how Google Search works”
Separating the three stages is not a presentation flourish, it is the main working tool. A page missing from the results failed at one precise stage, and the fixes that help at each stage have nothing to do with one another. Rewriting a page the engine never downloaded will produce nothing, however good the text. Adding links to a page that is properly indexed but judged less relevant than another will not settle the question either.
What is crawling, and why is it never complete?
Crawling is the downloading stage. A program travels the web, follows links, reads the addresses it has been given, and fetches the content of the pages it decides to visit. The important word is “decides”.
A discovered page is not necessarily crawled: some are disallowed by the site, others require authentication.
officialSource : Google Search Central, “In-depth guide to how Google Search works”
In other words, discovering an address and downloading it are two distinct events, and the second is not an automatic consequence of the first. On a small site the gap goes unnoticed. On a site that produces thousands of addresses from filter combinations it becomes the central question: an engine gives each site a limited share of attention, and a site that spends it on useless variants keeps none for its useful pages.
The robots.txt file is the standard lever for saying what you do not want downloaded. It is not
an informal convention: the protocol is a published specification.
A sitemap helps an engine discover URLs, but it commits to nothing about crawling them or indexing them.
officialSource : Google Search Central, “Learn about sitemaps”
A sitemap plays the mirror role: it points at addresses you would like discovered. It is useful where internal linking is thin, on a deep site, or for recent pages. It compensates for nothing else.
Declaring an XML sitemap
hypothesis
- Effort
- low
- Time to effect
- weeks
- Risk
- none
- Durability
- lasting
That card carries the “hypothesis” status, not “official”. What a sitemap does is documented; saying that the effort is low and the benefit lasting remains an editorial appraisal, which no document settles.
What is indexing, and what can fail there?
Indexing is the analysis and storage stage. The engine examines what it downloaded, tries to work out the subject, spots content that repeats content it already knows, and decides whether to keep the page. This is where the second most common illusion in the field lives.
Indexing is not guaranteed: not every page the engine processes ends up in the index.
officialSource : Google Search Central, “In-depth guide to how Google Search works”
That sentence is worth reading twice, because it comes from the engine itself and it contradicts a great deal of what is written elsewhere. A page can be perfectly reachable, technically flawless, fast, properly linked — and stay out. Nothing forces indexing, and services claiming otherwise are selling a coincidence.
The block that blocks nothing
Here is the case where intuition misleads most reliably, and it is explicitly documented:
A page disallowed in the robots.txt file stops the engine reading its no-index instruction, so the page can stay visible in the results.
officialSource : Google Search Central, “Block Search indexing with noindex”
The logic is simple once the three stages are held apart. The no-index instruction lives inside the page. Blocking access to the page means blocking the reading of the instruction. The engine then knows the address exists, because a link told it so, but it never received the instruction not to show it. The gesture meant to hide the page made the page impossible to hide.
It is also the best demonstration that “crawling” and “indexing” are not two words for the same thing. Anyone using them interchangeably will, sooner or later, produce exactly this mistake — and usually on the pages they most wanted taken down.
What is serving, and why can nobody describe it precisely?
The third stage picks, for a given query, what to show and in which order. It is the one people talk about most and know least.
What separates two close pages at the serving stage stays largely opaque from the outside, and the explanations in circulation are almost all reconstructions.
hypothesis
That “hypothesis” status is the most honest one available, and it will not turn into a stronger status by reading more articles. The systems are closed; the detailed explanations in circulation are reconstructions, sometimes sharp, never checkable from outside. What is documented, on the other hand, is that the result depends on the context of the query: the language, where you are searching from, the device. So there is no single results page, but a family of them — which is enough to explain why two rank trackers never agree.
What does not work, or no longer works
Three very common phrasings become hard to defend once the three stages are held apart.
Submit your site to the search engines to get indexed, and block in robots.txt the pages you do not want to appear.
The rest follows from the same reasoning. There is no standard “indexing delay” to announce, since the outcome itself is not settled. There is no single position to guarantee, since serving depends on context. And there is no technical setting that forces a stage the engine documents as discretionary.
What this split gives you instead is far more useful than a promise: a diagnostic method. Faced with a page that does not come up, one question comes first, and it is factual. Did the engine download this page? If not, the problem sits upstream, and the content will change nothing. If yes, is it kept? If not, the question becomes its own worth against what already exists. And if it is kept and still does not come up, then and only then does the discussion move to serving — that is, to the least documented ground of the three, where you advance with dated observations rather than with certainties.
Sources
- In-depth guide to how Google Search works — Google Search Centralaccessed on
- Block Search indexing with noindex — Google Search Centralaccessed on
- Introduction to robots.txt — Google Search Centralaccessed on
- Learn about sitemaps — Google Search Centralaccessed on
- RFC 9309, Robots Exclusion Protocol (September 2022) — IETFaccessed on
Block revalidated on
Frequently asked questions
- How long does it take for a new page to be indexed?
- No duration is published, and for a good reason: crawling depends on discovery, and indexing on a decision that is not guaranteed. A page may enter the index quickly, late, or never. Any promise of a precise delay is invented rather than measured.
- Does a sitemap get my pages indexed?
- No. It helps an engine discover addresses, which is useful on a large or poorly linked site. The official documentation states outright that it commits to neither crawling nor indexing. A sitemap solves a discovery problem, never a quality problem.
- How do I really stop a page appearing in the results?
- By letting the engine reach the page and serving it a no-index instruction. The opposite reflex is a trap: disallowing access in robots.txt stops that instruction being read, so the address can stay visible. Access and indexing are two separate settings.
- Why do two rank trackers never report the same position?
- Because the serving stage depends on the context of the query: language, device, location, the service's own history. So there is no single position to measure, but a family of results. A position figure is an estimate, not a reading.