Skip to content
search-engine-optimizedby Netsolution
  1. Home
  2. Files and configuration
  3. Robots.txt Generator

Robots.txt Generator

Produce a correct robots.txt without making the mistake that deindexes an entire site.

Generated locally following RFC 9309, with an up-to-date list of the publicly declared AI agents. Nothing is stored, and there is no usage limit.

What this tool is for

The robots.txt tells crawlers which parts of your site they may crawl. It fits in a few lines, sits at the root, and is the first file an engine consults on arrival.

Its apparent simplicity is misleading. It is one of the most dangerous files on the web: one misplaced line can remove an entire site from the results. The classic mistake is a Disallow: / inherited from staging and shipped to production.

The most widespread misunderstanding deserves saying up front: this file controls crawling, not indexing. Blocking a page here does not remove it from Google. Often the opposite happens.

To remember absolutely: to remove a page from the results you have to allow it to be crawled and put a noindex tag on it. If you block it here, Google cannot read the removal instruction — and the page stays indexed, with no title and no description.

How to read the results

User-agent designates the crawler concerned. The asterisk targets every crawler with no dedicated block. Careful: a crawler reads only the most specific block that concerns it, and then ignores the generic block entirely.

Disallow forbids a path. An empty value forbids nothing; a Disallow: / forbids the whole site. The difference is one character, and it is drastic.

Allow creates an exception inside a forbidden area. That is useful for blocking a folder while allowing a resource needed for display, such as a stylesheet.

Sitemap gives the full address of your site plan. It is the only directive independent of the user-agent blocks, and every engine reads it.

The priority rule is the longest path, not the order things are written in. That is counter-intuitive: always check the result with the robots.txt tester before publishing.

Common mistakes

Blocking a page to deindex it. The fundamental mistake. The result is a page that stays in the results with the note “no information available” — the worst of both worlds.

Leaving the staging Disallow in place. The staging site blocks everything, legitimately, and ships to production with the rule. It is the first thing to check when a new site does not index.

Blocking CSS and JavaScript files. An old practice that has become harmful: Google needs those resources to render the page and judge its mobile usability. Blocking them degrades its assessment.

Listing your sensitive areas in it. The file is public. Writing Disallow: /secret-admin/ amounts to publishing where your admin lives. Protect with authentication, do not advertise.

Forgetting the subdomains. Each needs its own file. The main domain’s does not apply to a blog hosted on a subdomain.

Going further

Frequently asked questions

Does blocking a page in robots.txt remove it from Google?

No, and it is the most expensive misreading in search. Forbidding crawling stops Google reading the page, but not indexing it if it discovers the address elsewhere. It then appears with no useful title or description. To remove a page you have to do the opposite: allow it to be crawled and put a noindex tag on it.

Do I need a robots.txt if I block nothing?

It is not required, but it is recommended. Its absence produces a 404 on every crawler visit, which is pointless. And it is the conventional place to declare your sitemap. A minimal file, allowing everything and pointing at the sitemap, is amply enough.

Does robots.txt really prevent access?

It is a convention, not a barrier. Serious crawlers respect it because their reputation depends on it; malicious scrapers ignore it. For genuinely sensitive content you need authentication or server-side blocking — never a line in a public file anyone can read.

Which rule applies when several match?

The most specific, that is the one with the longest path. An Allow on /private/public/ beats a Disallow on /private/. At equal length, the allow wins. It is counter-intuitive and it is the most frequent source of error.

Can I block AI crawlers here?

Yes, by targeting their agents: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and a few others. Mind the precision: blocking Google-Extended does not affect your ordinary indexing, but blocking Googlebot through imprecision would deindex the whole site.

Does the file have to be at the root?

Absolutely, and at the root of the exact domain. A robots.txt placed in a subfolder is never read. And every subdomain needs its own: the one on example.com does not apply to blog.example.com.