Screaming Frog SEO Spider
It is a crawler installed on your computer: it walks a site's addresses and hands back their response codes, their titles, their redirects and their links. It shows the structure as a robot meets it, and it measures no audience.
What it does
It starts from an address, follows the links it finds there, then starts again — exactly like a robot, except that it hands its inventory back in a table. For every address met: the HTTP response code, the title, the description, whether a canonical is present, the redirects passed through, the internal inbound and outbound links.
Its value comes down to one word: inventory. Most of a site’s technical faults are not visible page by page, but in a sorted list — the twenty addresses returning an error, the three-hop redirect chains, the identical titles across forty pages. Those are faults of distribution, and you have to see everything at once to spot them.
A crawler installed on a machine starts from your connection and your IP address, which makes it visible in the server logs as an unusual visitor and not as a search engine robot.
hypothesis
What it does not do
This is the most useful section of the entry, and the reason it exists.
A crawler shows what the site exposes, never what the engine kept of it: the two lists always differ, and only the second comes from the engine.
hypothesis
It does not say what is indexed. It does not say what receives visits, or on which queries. It estimates no search volume. It does not judge the quality of a text. And it does not reproduce a search engine robot’s behaviour: it crawls by its own rules, from your machine.
What it shows is therefore the site’s state of exposure, not its state in an index. Confusing the two leads to spending a long time fixing pages the engine was not concerned with.
What it costs
licence, one to four seats : €245 per year
The paid version is an annual licence per seat, whose unit price decreases in volume tiers.
officialSource : Screaming Frog, SEO Spider pricing page
A free version exists, whose crawling is capped at a number of addresses announced on the vendor's pricing page.
officialSource : Screaming Frog, SEO Spider pricing page
The model is worth noting because it has become rare: an annual licence per seat, and not a subscription to an online service. That has a practical consequence appearing on no grid: the data stays on your machine, and nothing is interrupted the month you stop paying — the licence expires, the tool falls back on its limited version.
What replaces it without paying
The same tool’s free version, first, as long as the site fits under its cap. It is the most honest replacement: it is the same software.
Then two open tools, each taking over part of it: advertools redoes the crawl with no cap, but as a script and with no interface; Lighthouse redoes the audit, page by page, and crawls nothing. Neither replaces the immediate sorting of a complete table, which is what the licence actually buys.
Then, for a different need, the engine’s Search Console. It does not do the same thing — it says what the engine kept, not what the site exposes — but if you have to pick one source of information about a single site, it is that one, and it is free.
Crawling your own site before concluding anything at all
hypothesis
- Effort
- low
- Time to effect
- immediate
- Risk
- none
- Durability
- structural
Affiliation
This site takes part in no affiliate programme for this tool, and the order of this category depends on no payment. The price displayed above was recorded on the vendor’s page, on the date shown beside the amount, and nowhere else.
What does not work
Run a full crawl, fix every error the tool reports, and your site will finally be clean in Google's eyes.
Sources
- SEO Spider — Pricing — Screaming Frogaccessed on
- About Search Console — Google Search Consoleaccessed on
Block revalidated on
Entries in this category
Frequently asked questions
- Is the free version enough for a small site?
- Often yes, as long as the site stays under the address cap the vendor announces. A brochure site or a blog of a few dozen pages fits well inside it, and a full crawl reveals exactly the same faults there as on a large site.
- Does this tool say whether my pages are indexed?
- No, and that is the most frequent confusion. It shows what your site answers when walked. What the engine actually kept is read in Search Console, free, and the two inventories never quite coincide.
- Should it be installed on a server or on your own machine?
- It is desktop software: it starts from your machine and your connection. On a large site the crawl therefore consumes your bandwidth and appears in the server logs as a very active visitor, which is better warned about in advance.
- When does a crawler become genuinely necessary?
- When the number of pages exceeds what can be checked by hand, or after a migration, when you have to prove that no old address returns an error. Before that, it mostly shows problems you already knew about.