Skip to content
search-engine-optimizedby Netsolution

Screaming Frog SEO Spider

It is a crawler installed on your computer: it walks a site's addresses and hands back their response codes, their titles, their redirects and their links. It shows the structure as a robot meets it, and it measures no audience.

What it does

It starts from an address, follows the links it finds there, then starts again — exactly like a robot, except that it hands its inventory back in a table. For every address met: the HTTP response code, the title, the description, whether a canonical is present, the redirects passed through, the internal inbound and outbound links.

Its value comes down to one word: inventory. Most of a site’s technical faults are not visible page by page, but in a sorted list — the twenty addresses returning an error, the three-hop redirect chains, the identical titles across forty pages. Those are faults of distribution, and you have to see everything at once to spot them.

A crawler installed on a machine starts from your connection and your IP address, which makes it visible in the server logs as an unusual visitor and not as a search engine robot.

hypothesis

What it does not do

This is the most useful section of the entry, and the reason it exists.

A crawler shows what the site exposes, never what the engine kept of it: the two lists always differ, and only the second comes from the engine.

hypothesis

It does not say what is indexed. It does not say what receives visits, or on which queries. It estimates no search volume. It does not judge the quality of a text. And it does not reproduce a search engine robot’s behaviour: it crawls by its own rules, from your machine.

What it shows is therefore the site’s state of exposure, not its state in an index. Confusing the two leads to spending a long time fixing pages the engine was not concerned with.

What it costs

licence, one to four seats : €245 per year(checked on)

The paid version is an annual licence per seat, whose unit price decreases in volume tiers.

officialSource : Screaming Frog, SEO Spider pricing page

A free version exists, whose crawling is capped at a number of addresses announced on the vendor's pricing page.

officialSource : Screaming Frog, SEO Spider pricing page

The model is worth noting because it has become rare: an annual licence per seat, and not a subscription to an online service. That has a practical consequence appearing on no grid: the data stays on your machine, and nothing is interrupted the month you stop paying — the licence expires, the tool falls back on its limited version.

What replaces it without paying

The same tool’s free version, first, as long as the site fits under its cap. It is the most honest replacement: it is the same software.

Then two open tools, each taking over part of it: advertools redoes the crawl with no cap, but as a script and with no interface; Lighthouse redoes the audit, page by page, and crawls nothing. Neither replaces the immediate sorting of a complete table, which is what the licence actually buys.

Then, for a different need, the engine’s Search Console. It does not do the same thing — it says what the engine kept, not what the site exposes — but if you have to pick one source of information about a single site, it is that one, and it is free.

Crawling your own site before concluding anything at all

hypothesis

Effort
low
Time to effect
immediate
Risk
none
Durability
structural

Affiliation

This site takes part in no affiliate programme for this tool, and the order of this category depends on no payment. The price displayed above was recorded on the vendor’s page, on the date shown beside the amount, and nowhere else.

What does not work

Run a full crawl, fix every error the tool reports, and your site will finally be clean in Google's eyes.

A crawler reports deviations from its own rules, many of which have no known consequence. Nothing it displays says what the engine kept or what it ignores: that is read in Search Console, which the engine's vendor publishes free.

Sources

  1. SEO Spider — Pricing — Screaming Frogaccessed on
  2. About Search Console — Google Search Consoleaccessed on

Block revalidated on

Entries in this category

Frequently asked questions

Is the free version enough for a small site?
Often yes, as long as the site stays under the address cap the vendor announces. A brochure site or a blog of a few dozen pages fits well inside it, and a full crawl reveals exactly the same faults there as on a large site.
Does this tool say whether my pages are indexed?
No, and that is the most frequent confusion. It shows what your site answers when walked. What the engine actually kept is read in Search Console, free, and the two inventories never quite coincide.
Should it be installed on a server or on your own machine?
It is desktop software: it starts from your machine and your connection. On a large site the crawl therefore consumes your bandwidth and appears in the server logs as a very active visitor, which is better warned about in advance.
When does a crawler become genuinely necessary?
When the number of pages exceeds what can be checked by hand, or after a migration, when you have to prove that no old address returns an error. Before that, it mostly shows problems you already knew about.