Skip to content
search-engine-optimizedby Netsolution

Crawling and technical audit

A crawling tool walks your site the way a crawler would and reports what it meets: redirects, unreachable pages, missing titles, contradictory instructions. It shows the site's real structure, often different from the one you thought you built, but it judges no page.

What does a crawling tool do?

It starts from an address, follows the links it finds there, and starts again. At the end it hands over the list of pages reachable from that entry point, with what it met on the way: response codes, redirect chains, titles, indexing instructions, duplicate addresses.

Its value lies in a shift of viewpoint. You know the site you meant to build; the tool shows the one a crawler meets. The gap between the two is nearly always wider than expected, and it is made of reachable pages you thought were removed, stacked redirects, and address variants produced by filters.

A crawler does not download every page it discovers: some are disallowed by the site, others require authentication.

officialSource : Google Search Central, “In-depth guide to how Google Search works”

Which problem does this category detect best?

This one, and it justifies the exercise on its own:

A page disallowed in the robots.txt file stops the engine reading its no-index instruction, so it can stay visible in the results.

officialSource : Google Search Central, “Block Search indexing with noindex”

A configuration where access is disallowed while a no-index instruction waits to be read is invisible to the naked eye: each file, taken separately, looks correct. A tool reading both at once flags it immediately. It is the category’s most profitable use case, and it mostly concerns sites that have just moved from staging to production.

Crawling your site before a deployment that touches addresses

hypothesis

Effort
moderate
Time to effect
immediate
Risk
none
Durability
structural

What a crawling tool does not do

It does not say whether a page deserves indexing, and it never will:

Indexing is not guaranteed: not every page the engine processes ends up in the index.

officialSource : Google Search Central, “In-depth guide to how Google Search works”

Nor does it see what the engine actually did with your pages. A crawl is a simulation run by a robot that is not the engine’s, at a given moment, from a starting point you chose. It shows what is reachable, not what was kept.

Finally, it does not rank its findings reliably.

Most alerts produced by an automated audit flag a deviation from a convention internal to the tool, and not an observed obstacle to crawling or indexing.

hypothesis

That “hypothesis” status is owned up to: it is an observation from practice, not a documented property of the tools. It has an immediate practical consequence, which is to read the alert list looking for the three categories that genuinely prevent something — access refused, contradictory instruction, unstable address — and to ignore the rest until those three are dealt with.

What replaces a crawling tool?

For a small site, Search Console is enough to answer the most useful question: was this page crawled, and is it kept? It answers from the engine’s point of view, which no third-party tool can do. It does not give a crawl’s overall view, and that is exactly why the two do not compete.

To check an exclusion instruction, the protocol itself is public and standardised: the syntax can be read straight from the specification, with no intermediary.

What does not work, or no longer works

Your technical score is 62 out of 100: fix the 214 errors detected to improve your ranking.

The score and the error count depend on which rules are switched on in the tool, not on the site's state. No engine publishes a technical score, and nothing connects a list of conventional deviations to a change of position.

Five entries are published here. Two paid tools: Screaming Frog SEO Spider, which hands back the raw inventory of addresses, and Sitebulb, which ranks its findings for you — which is a convenience and one more interpretation.

Three open tools each take a share of that: advertools redoes the crawl as a script, with no interface; Lighthouse audits one page at a time; Unlighthouse takes that same audit to the scale of a site, by sampling.

None of the three hands back the raw inventory, which is what the paid licence actually buys. The page open source tools, in French, details that split.

Sources

  1. In-depth guide to how Google Search works — Google Search Centralaccessed on
  2. Block Search indexing with noindex — Google Search Centralaccessed on
  3. Introduction to robots.txt — Google Search Centralaccessed on
  4. RFC 9309, Robots Exclusion Protocol (September 2022) — IETFaccessed on

Block revalidated on

Entries in this category

Frequently asked questions

At what site size does a crawling tool become useful?
As soon as you can no longer list your pages from memory, which happens sooner than you think on a site with filters or pagination. Below that, Search Console and a careful reading of the sitemap are enough to spot the same problems.
Does an audit returning hundreds of alerts reveal a site in bad shape?
Not necessarily, and that is this category's trap. The number of alerts depends mostly on how many rules are switched on in the tool, not on the site's real state. Three alerts on a forgotten no-index instruction count more than two hundred remarks about title lengths.
Should you crawl your site before every deployment?
Before deployments touching addresses, redirects or indexing instructions: yes, systematically. Those are precisely the changes that break silently, showing nothing abnormal on screen. For a simple text edit, the exercise adds nothing a careful reread does not.
What can be replaced with free tools in this category?
Search Console reports the crawl and index state of your pages as the engine sees them, which no third-party tool can know. It does not replace a full crawl, but it answers the most useful question first.