Skip to content
search-engine-optimizedby Netsolution
  1. Home
  2. Files and configuration
  3. Robots.txt Tester

Robots.txt Tester

Check, URL by URL, what your robots.txt really allows.

Your robots.txt is fetched live and interpreted under RFC 9309, reproducing the priority logic Google applies. No usage limit.

What this tool is for

A robots.txt looks simple to read, and rarely is. The priority rule is not the order things are written in but the length of the path, wildcards complicate matters, and each crawler reads only the block that concerns it.

The result: you think you have blocked a whole section and a sub-part stays open. Or the reverse, which is worse — you think everything is open and an inherited rule blocks the pages that matter.

This tester settles the question. You give a URL, it tells you whether it is allowed for the chosen crawler, and it displays the exact rule that produces that result. No interpretation, one answer.

When it is indispensable: before putting a change to the file live. A robots.txt mistake can remove an entire site from the results within days, and recovery takes far longer than the fall.

How to read the results

The allowed or blocked verdict answers the question asked: can this crawler reach this address?

The rule applied is the most useful piece. It displays the exact line that decides, which lets you understand why the result differs from your intention.

The user-agent block used says which group of rules applied. A frequent surprise: a crawler with its own block ignores the generic block entirely, including the rules you thought were universal.

The comparison between crawlers reveals differences in treatment. That is where you discover a section is open to Google and closed to AI crawlers, or the reverse — the AI crawler access checker goes further on that point.

The indexing warning appears when a blocked URL can nevertheless be indexed. It is the reminder that blocking does not mean removal.

Common mistakes

Blocking in order to deindex. The fundamental misunderstanding, which produces results with no title and no description. Allow crawling and use noindex.

Reasoning from the order of the lines. It is the length of the path that decides, not the position in the file. Many configurations believed correct are not.

Testing only with Googlebot. If you have defined specific blocks, each obeys its own rules. Compare at least the agents that matter to you.

Testing paths rather than real URLs. Query parameters sometimes change the result. Test the addresses as they exist, parameters included.

Changing the file without testing afterwards. Google caches it for about twenty-four hours: a mistake published on a Friday evening acts all weekend before anyone even notices.

Going further

Frequently asked questions

Why does a blocked URL still appear in Google?

Because blocking prevents crawling, not indexing. If Google discovers the address through an external link, it can index it without ever reading the page — and display it with no useful title, carrying the note that no information is available. That is the least desirable result possible.

Which rule wins when several match?

The one with the longest path, regardless of the order they are written in. An Allow on /folder/page.html beats a Disallow on /folder/. At equal length, the allow wins. That rule surprises people, which is why testing beats reasoning.

Are wildcards recognised?

Yes by Google and Bing, but they are not part of the original standard. The asterisk stands for any run of characters, the dollar marks the end of the URL. A Disallow on /*.pdf$ blocks every PDF. Other, more rudimentary crawlers do not interpret them.

Does the test hold for every crawler?

No, and that matters. Each crawler reads only the most specific user-agent block that concerns it, ignoring the generic block entirely. A URL allowed for Googlebot can be blocked for GPTBot, and the other way round. Test every agent you care about.

How do I properly remove a page from the results?

By allowing it to be crawled and putting a noindex tag on it, or an X-Robots-Tag header. Google has to be able to read the instruction to apply it. Once the page is deindexed you can block it if you wish — but not before.

Is my robots.txt taken into account immediately?

Google caches it for about twenty-four hours. A change is therefore not applied within the minute. In an emergency, Search Console lets you force a re-read of the file.