- Home
- Artificial intelligence
- AI Crawler Access Checker
AI Crawler Access Checker
Do you know whether AI systems are allowed to read your site? Most sites do not.
Direct analysis of your robots.txt file, tested against the list of agents publicly declared by OpenAI, Anthropic, Google, Perplexity, Meta, Amazon and Common Crawl. Nothing is stored, and there is no usage limit.
What this tool is for
A growing share of searches no longer goes through a list of blue links. A question is put to ChatGPT, to Perplexity, or to the summaries generated by Google, and back comes a written answer citing a few sources. Being one of those sources is becoming a channel in its own right.
Those systems only reach your site if your robots.txt lets them. And most sites have never settled the question: their file predates the existence of these crawlers, so it does not mention them. By default, the absence of a rule means permission — you are probably open without knowing it.
This tool tests your file against every known agent and tells you, crawler by crawler, whether the door is open, closed, or simply not mentioned. You will finally know your real situation, and be able to decide instead of having it decided for you.
Why now is the moment: the question is new, and most sites have decided nothing yet. Taking a position now, either way, puts you ahead of those who will discover the subject in two years.
How to read the results
Each crawler falls into one of three states, and the nuance matters.
Allowed means no rule blocks it. Careful: that is also the result when your file does not mention the crawler at all. Allowed by choice and allowed by oversight produce the same effect, but not the same control.
Blocked means a directive explicitly targets that agent. The tool displays the exact line responsible, which lets you check that it does what you believed.
Partially blocked signals that only certain sections are forbidden. That is often the wisest configuration: open your reference content, protect your commercially valuable pages.
The agents are not interchangeable, and confusing them is expensive. GPTBot feeds the training of OpenAI’s models, while OAI-SearchBot serves live search inside ChatGPT — the one that cites you. Blocking the first without the second is often the right compromise. Likewise, Google-Extended concerns use by Google’s AI systems, and has no effect on your classic indexing by Googlebot.
Common mistakes
Blocking everything as a protective reflex. Forbidding every AI agent removes you from generated answers, which already account for a significant share of searches. If your model rests on visibility rather than advertising, you are closing a channel for nothing.
Blocking Googlebot while meaning to target AI. It is the most serious mistake possible, and it happens: a badly written rule deindexes the site from classic search. Always check the exact reach of your directives with the robots.txt tester before publishing.
Believing a robots.txt really prevents access. It is a convention of use, respected by players with a reputation to defend, ignored by the rest. For genuinely sensitive content you need authentication or server-side blocking — not a line in a public file.
Opening the door without preparing the house. Allowing access is not enough to be cited. Models take up short, factual, attributable passages. Diffuse content, with no structure and no identifiable source, will be read and never used. The llms.txt generator helps present your important pages in a usable form.
Configuring once and never coming back. New agents appear regularly, each arriving with a name your file does not know. A check every six months is enough to stay in control.
Going further
- llms.txt GeneratorBuild the llms.txt file that tells AI assistants what your site is about
- Robots.txt TesterTest whether a URL is allowed or blocked, and by which rule
- Googlebot SimulatorSee a page exactly as a search engine crawler sees it
Frequently asked questions
Should I allow or block AI crawlers?
It depends on your business model, and there is no universally right answer. If you sell a service and want visibility, being cited by ChatGPT or Perplexity brings you customers: allow them. If you live off advertising or sell access to your content, an AI answering in your place deprives you of the visit: block them. The trap is having that choice made for you.
Does blocking GPTBot remove me from ChatGPT?
Not immediately, and not entirely. GPTBot feeds model training; OAI-SearchBot serves live search. Blocking the first does not stop the second citing you. And a model already trained on your older pages will go on knowing them. Blocking acts on the future, not the past.
Does blocking AI crawlers hurt my Google rankings?
No, provided you target the right agents. Googlebot indexes for classic search; Google-Extended concerns use by Google’s AI models. You can block Google-Extended and stay perfectly indexed in search. The serious mistake would be blocking Googlebot through imprecision in your robots.txt.
Is robots.txt actually respected?
It is a convention, not a technical barrier. The major players — OpenAI, Anthropic, Google, Perplexity — publish their agents and respect them, because their reputation depends on it. Anonymous scrapers ignore it. If you genuinely need to prevent access, you need server-side blocking by agent or IP address, not a line in a text file.
What is llms.txt and should I have one?
It is a file at the root of the site that presents your content to language models in condensed form, listing your important pages. It is not an official standard and adoption remains partial. The cost of setting it up is low and the risk nil: a reasonable bet if your visibility in AI systems matters to you.
How do I know whether AI systems actually cite me?
Access and citation are two different things. This tool checks that the door is open, which is necessary but not sufficient. To be cited you also need content structured as short factual answers, checkable claims, and a clearly identified source. Content an AI can read but does not understand will not be taken up.