Skip to content
search-engine-optimizedby Netsolution

advertools

It is an open library, not software you click: you write a few lines to crawl a site, read a robots.txt file or a sitemap, and you get a table back. It replaces part of a paid crawler, at the price of the script.

What it does

It crawls a site by following its links and hands the inventory back as a table: response codes, titles, descriptions, canonicals, internal links, and any extra elements you explicitly ask for. It also reads robots.txt files and sitemaps, which makes it the shortest tool for answering “what does this site declare it exposes?”. Text analysis and advertising campaign functions come on top, outside the scope of this entry.

The real benefit is not the feature list — a paid crawler does as much. It is reproducibility: a script reruns identically six months later, on another site, and produces a table comparable with the previous one. No setting clicked by hand offers that guarantee.

The library is published under the MIT licence, declared in the public repository hosting its code.

officialSource : advertools, public repository and licence file

Development is under way: the public repository carries changes dated June 2026, less than two months before the check.

officialSource : advertools, repository commit history

That second check is not a formality. An abandoned open project stays installable and simply stops being repaired — on a tool that depends on what sites and engines return, abandonment is a deferred failure. The date of the last published work is looked at before adopting anything open, this one as much as any other.

What it does not do

A library has no interface: it only replaces click-driven software for someone who can write the script, and the cost it removes in licensing comes back as time spent writing and rereading.

hypothesis

There is no screen, no column to sort with a click, no report to send as it stands. Spotting “the forty pages with an identical title” needs one more line of code, where a commercial tool displays it by default. On a one-off audit, that line costs more than the licence it avoids; on a check repeated twelve times a year, it costs infinitely less.

The functions that query a search engine assume access supplied by that engine or by a third party, separate from the library: the code is free, the search data is not.

hypothesis

And like any crawler, it shows what the site exposes, never what the engine kept. The two inventories overlap without coinciding, and only the second comes from the engine.

What it costs

Nothing in licensing, and no paid hosted offer from the author has been recorded. The cost is entirely elsewhere: knowing how to write the script, knowing how to reread it, and accepting that it has to be revisited when the library changes. It is a cost in skill, which does not transfer and cannot be bought at the moment you need it.

When the paid tool stays the right choice

When the result has to be read by somebody who did not produce it. A raw table is an excellent working instrument and a poor deliverable: the paid crawler in this section — Screaming Frog SEO Spider — sells formatting, immediate sorting and a transmissible report first.

The right split is therefore not “free versus paid” but one-off versus repeated. A single audit is faster with a mouse. A monthly check, on the same points, is better done as a script — and above all it actually gets done, which is never the case with manual checks.

Scripting the technical checks you redo periodically

hypothesis

Effort
high
Time to effect
weeks
Risk
none
Durability
structural

Affiliation

This site takes part in no affiliate programme for this tool, and none exists: an open library pays nobody to be recommended. This entry earns nothing, and the order of this section has never depended on payment.

What does not work

A free open source alternative that completely replaces your paid crawler, with no URL limit at all.

The sentence compares software and a library as though they were the same object. What you pay for in a commercial crawler is not permission to crawl, it is the interface, the immediate sorting and the transmissible report — precisely what a library does not supply. The address cap disappears; it is replaced by a skill cap, which appears on no pricing grid.

Sources

  1. advertools — public repository, licence and history — Elias Dabbasaccessed on
  2. About Search Console — Google Search Consoleaccessed on

Block revalidated on

Entries in this category

Frequently asked questions

Do you need to be able to program?
Yes, a little Python. That is the entry condition, and it is disqualifying: a few lines are enough to crawl a site, but you have to know how to write them, rerun them and read a table of results. Without that, a click-driven crawler costs less in time.
What does it replace of a paid crawler?
The crawling itself, reading robots.txt files and sitemaps, extracting chosen elements from pages. What it does not replace: the interface that sorts and filters effortlessly, and the ready-made reports a commercial tool delivers in one click.
Does it measure positions in the results?
It can query a search engine, but access to the results is supplied separately and does not come with the library. The code is open; search data remains a service that is requested, authorised and most often paid for by volume consumed.
Is it suitable for a very large site?
That is where it takes the advantage, because a script reruns identically, can be scheduled and hands back a table comparable from one run to the next. A desktop crawler by contrast asks a human to reproduce the same settings every time.
Does it say what the engine indexed of my site?
No, no crawler does. It shows what your site exposes when walked. What the engine actually kept is read in its console, free and for sites whose ownership is proven, and the two inventories always differ.