- Home
- Technical and performance
- Googlebot Simulator
Googlebot Simulator
See your page as a crawler discovers it, before any JavaScript runs.
The request is made with the chosen user agent, with no JavaScript executed, respecting your robots.txt. That is a crawler’s behaviour on its first pass. No usage limit.
What this tool is for
A crawler does not see what you see. It does not click, does not wait for animations, and on its first pass does not execute JavaScript. It reads a text document and extracts what it can.
The gap with the real display can be total. On a site built entirely on the client, that initial document sometimes comes down to an empty tag and a script call. The page is beautiful in a browser and empty to a crawler.
This simulator reproduces that first pass and shows you what comes out of it: the content genuinely accessible, the hierarchy of headings, the links a crawler can follow, and the indexing directives it receives.
The test that settles it: look for a specific sentence from your main content in the result. If it is not there, your indexing depends on a second pass whose timing and occurrence are both unguaranteed.
How to read the results
The accessible content is the main output. It is the text a crawler can read without executing a single script. It should contain the substance of your page.
The heading structure reveals the hierarchy as perceived. One H1, H2s that divide the page logically, subordinate H3s. A page with no H1, or with six, transmits a confused structure.
The followable links are the ones a crawler can take to discover other pages. Links created by JavaScript do not appear: that is how whole sections of a site become unreachable to crawling.
The indexing directives gather the robots tag, the X-Robots-Tag header and the canonical. This is where forgotten noindex tags appear, the leading cause of non-indexing.
The structured data confirms your eligibility for rich results. Its absence is not a fault, but it is a missed opportunity.
Common mistakes
Assuming Google will see the content anyway. It may see it, later. In the meantime your page is indexed empty. On a site that publishes regularly, that lag is a permanent handicap.
Building the navigation in JavaScript. A menu that does not produce real link tags with an href attribute makes whole sections undiscoverable. It is one of the most expensive and most frequent faults.
Loading content on scroll with no alternative. The crawler does not scroll. Anything that appears only after an interaction is invisible. Provide classic pagination alongside.
Testing only with the Googlebot agent. AI crawlers are stricter and generally execute no JavaScript. If their visibility matters to you, compare — the AI crawler access checker completes this analysis.
Forgetting to check after a rebuild. Moving to a modern development framework often radically changes what a crawler receives, with nothing changing on screen.
Going further
- Robots.txt TesterTest whether a URL is allowed or blocked, and by which rule
- AI Crawler Access CheckerCheck whether GPTBot, ClaudeBot and PerplexityBot can read your pages
- Meta Tags AnalyzerTitle, description, Open Graph and indexing directives of a page
Frequently asked questions
Why does my content not appear in the result?
Because it is produced by JavaScript in the browser. The simulator reproduces a crawler’s first pass, which does not execute JavaScript. Google does, but on a second visit with a variable delay and no guarantee. Content absent from this result is content whose indexing is uncertain.
But Google does execute JavaScript, doesn’t it?
Yes, but in two stages. It first crawls the raw HTML, queues the page for later rendering, then comes back. That delay runs from a few hours to several weeks depending on the resources allocated to your site. Other engines and most AI crawlers do not do it at all.
What is the point of changing the user agent?
To detect differential treatment. Some servers, often through an application firewall, block crawlers or serve them a degraded version. Comparing the response obtained as a browser and as Googlebot reveals that kind of configuration immediately.
Does the simulator respect my robots.txt?
Yes, and it tells you explicitly if the requested URL is forbidden by your own rules. That is valuable information in itself: discovering that an important page is blocked by a forgotten directive explains many missing indexations.
What is structured data and why check it here?
They are JSON-LD blocks describing the content in a machine-readable way: an article, a product, a recipe, an FAQ. They govern your eligibility for rich results. Since they sit in the HTML, this simulator confirms that a crawler sees them on the first pass.
How does this differ from fetching the source code?
Fetching the source gives you the raw HTML to read yourself. The simulator interprets that document from a crawler’s point of view: it extracts the heading structure, the followable links, the indexing directives and the structured data. One shows, the other analyses.