There’s a comfortable assumption that most web teams carry around: if the site works in the browser, it works for SEO. The page loads, the content appears, users are happy — job done.
This assumption is dangerous. With AI and large language models are increasingly shaping how content is discovered and surfaced, it’s becoming more dangerous by the day.
The reality is that what your users see and what bots see can be entirely different things. And if bots can’t see your content, they can’t rank it, retrieve it, or include it in AI-generated answers.
The Gap Between What Users See and What Bots See
To a user, a modern JavaScript-heavy website looks perfectly functional. Images load, text appears, and interactive elements respond. But strip away JavaScript — which is precisely what many crawlers and all basic LLM scrapers do —, and you’re often left with a blank page, a loading spinner, or a skeletal HTML shell with no meaningful content at all.
Sites that rely on client-side rendering can experience indexation rates up to 40% lower than those of server-rendered equivalents. That’s not a small edge case. That’s a significant slice of your content that search engines may never properly index.
The gap matters now more than ever because we’re no longer just optimising for one type of bot. We’re optimising for search engine crawlers and for the scrapers that feed LLM training data and retrieval-augmented generation (RAG) systems — and these two audiences have very different capabilities.
How Search Engines Actually Process Pages
To understand why rendering matters so much, consider what happens when Googlebot visits your site. The process runs through four stages:
- Crawl queue — a prioritised repository of URLs awaiting discovery.
- Crawler — Googlebot fetches the HTML and resources from your server.
- Processing — initial analysis of HTML structure and links.
- Index — final storage in the searchable database.
For a plain HTML page, that’s a fairly clean pipeline. But for a JavaScript-heavy site, there’s a detour.
If the processing stage detects heavy JavaScript dependency, the URL gets sent to a separate render queue. Google’s Web Rendering Service then executes the JavaScript in a headless Chromium environment, waits for the final DOM to materialise, and sends the rendered HTML back for re-processing. Only then does the content make it into the index.
This creates three critical constraints:
1. Rendering is deferred
Unlike static HTML, rendering happens in a “second wave.” There’s a real time lag (sometimes days or weeks) between a page being discovered and its content becoming fully visible to Google.
2. Rendering is resource-intensive
Executing JavaScript requires significant CPU and memory overhead. Google scales this based on a site’s crawl budget, meaning lower-authority sites are at greater risk of having pages skipped or only partially rendered.
3. Rendering is selective and partial
To conserve resources, Googlebot may time out scripts, skip non-critical files, or fail to render pages that exceed its thresholds. Your content might be partially indexed — or not at all.
It’s worth acknowledging: Google has got considerably better at handling JavaScript over the years. For a typical well-built site, it’s no longer the catastrophic SEO blocker it once was.
But the same absolutely cannot be said for AI and LLMs.
How LLMs Process Pages Differently — and Why JS Is a Problem
Large language models retrieve and process web content in a fundamentally different way from traditional search engines. Understanding this architecture is essential for anyone thinking about visibility in an AI-driven search world.
When an LLM-powered search system receives a query — say, “When is the World Cup?” — it doesn’t just look up a URL in an index. The system works roughly like this:
- Knowledge base / document retrieval — the crawled and stored content is pulled from a knowledge base.
- Chunking and embedding — documents are broken into chunks and converted into vector representations.
- Semantic vector index — those vectors are stored in a database indexed by meaning, not by keyword.
- Prompt augmentation — the user's query is enriched with the retrieved semantic data.
- LLM generation — the augmented prompt is fed into the model, which synthesises a single, comprehensive response.
This is a fundamentally different architecture from traditional search, with three key differences:
1. Vector embeddings replace raw HTML crawling
Documents need to be fully readable text before they can be chunked and vectorised. If your content is injected by JavaScript after page load, and the scraper can’t execute that JavaScript, the content doesn’t exist to the embedding pipeline.
2. The semantic vector index searches by meaning, not keywords
This is actually good news for well-structured content, but only if that content makes it into the index in the first place.
3. Prompt augmentation requires complete context
The system builds an enriched prompt from what it retrieved. If it only retrieved a loading spinner, your brand won’t feature in the answer.
How JavaScript Creates Problems for LLMs
Crawl and fetch issues
Most basic scrapers and crawlers used to feed LLM training data or RAG systems don’t have a built-in browser engine. They fetch raw HTML and move on. If your content is injected via JavaScript, the bot initially (and often permanently) sees a blank page. If the data isn’t in the raw HTML, it effectively doesn’t exist to these systems.
Resource-intensive rendering
Rendering a JavaScript-heavy page requires a full headless browser to execute scripts, download secondary payloads, and wait for the final DOM to render. This is around 100 times more computationally expensive than reading flat HTML.
For Google, they may defer this for days or weeks.
For LLMs, they will simply time out if your JavaScript takes more than a few seconds to execute — leading to incomplete retrieval and, ultimately, hallucinated answers about your content.
Chunking is problematic
LLMs need the full text of a page in order to break it into meaningful chunks for vectorisation. If your site uses infinite scroll or “click to load” patterns, the retriever only ever sees the first few paragraphs. And if that incomplete context is all that ends up in the semantic vector index, users asking relevant questions simply won’t find you.
It creates what you might call a knowledge gap in the vector space, that is, a space where your content should exist but doesn’t.
Fragmented DOM adds noise
Modern JavaScript frameworks like React and Vue often create complex, deeply nested DOM structures. When an LLM tries to understand a page, it has to filter through all of that code. Excessively complex JS-generated structures confuse the retriever module, making it harder for the AI to distinguish between your actual content and navigation menus, tracking scripts, or advertising.
The consequence of all of this is stark:
If content is incomplete → it hallucinates.
Not All Rendering Approaches Are Equal
The solution to all of this is rendering, but there are important considerations to keep in mind when making these decisions.
There are four main rendering strategies, each with different implications for SEO, performance, and AI visibility.
Client-Side Rendering (CSR)
The server sends HTML, CSS, and JavaScript to the browser. The browser then dynamically generates the content in the DOM. The user sees a fully rendered page. The bot sees very little.
CSR is the worst option for both traditional SEO and AI discoverability. Avoid relying on it solely for any content you want indexed.
Image Credit: Debugbear
Server-Side Rendering (SSR)
The server does the heavy lifting. It generates the complete HTML for a page before sending it to the browser. The browser displays the content immediately on arrival, so it is seen straight away by the bots.
SSR is the recommended approach if you want your site to be both search-ready and AI-ready.
When a bot knocks on the door, the content is already sitting there in plain text with no assembly required.
Image Credit: Debugbear
Static Site Generation (SSG)
Content is pre-built into static HTML, CSS, and JavaScript files at build time rather than on demand. The result is a blazing-fast, highly crawlable site.
SSG is excellent for SEO but works best for content that doesn’t change frequently. A blog or brochure site is ideal; a live product catalogue is less so.
Image Credit: IONOS
Incremental Static Regeneration (ISR)
A hybrid of SSG and SSR. Pages are pre-built as static files but can be regenerated on demand after deployment, without rebuilding the entire site. It offers the speed of static with more manageable freshness.
ISR is a strong choice for larger sites that need the SEO benefits of static rendering without the staleness problem.
Image Credit: Smashing Magazine
Diagnosing Rendering Issues
Now, all that is well and good, but I’ve found that most teams don’t test what bots are actually rendering.
Here are some free tools to start testing rendering so you can get started today. While these tools primarily focus on Googlebot, it is safe to say that if Googlebot is struggling to render content, it will almost certainly be an issue for LLMs.
Google Search Console URL Inspection Tool
The URL Inspection tool in GSC is your primary resource for checking how Google saw an indexed page.
- Enter a URL and check its indexing status.
- Click “View Crawled Page” to see the HTML as it was fetched, along with details of how Google rendered it during the last successful crawl.
- Inspect the HTML code directly to look for missing content.
Pro tip: Copy and paste the entire “View Crawled Page” HTML into a new .html file and open it in your browser. This gives you a close approximation of how complete Googlebot’s final output was. If content or styling is missing, Googlebot likely struggled with rendering.
URL Inspection Live Test
Staying within Google Search Console, the Live Test provides a current, standalone render of the page.
- Run the Live Test to fetch and render the current version of the page.
- Check the “View Tested Page” section — this is often more complete than the crawled view, though less representative of a real crawl.
- Look at the screenshot: a blank, incomplete, or broken screenshot is a strong signal that rendering failed.
- Check the “More Info” tab for page resources — anything that failed to load is a potential cause of rendering failure.
- If the Live Test returns a generic “Something went wrong…” error, the page is likely too heavy or too slow to render.
PageSpeed Insights
While PageSpeed Insights is primarily a performance tool, its metrics provide excellent clues about rendering risk.
- If the page takes 5–7 seconds or more to fully load, it is at high risk of rendering issues.
- If it takes over 10 seconds, that’s a serious danger.
- Total resource size over 500KB is approaching a danger zone.
- Resources that are several megabytes in size are a red alert.
Slower and heavier pages are exponentially more likely to suffer rendering failures — both for search engines and, even more so, for LLMs.
Designing for Bots Without Harming UX
A common concern: “I don’t know how to design a site for bots without harming the user experience.”
The good news is that the two goals are not in conflict. The principle is simple:
Think of it like building a car: HTML gives you the structure and the body, CSS gives you the paint and the style, and JavaScript gives you the engine and the interactivity.
The car should exist and be usable without the engine running at full speed. The content — the thing bots need — should live in the HTML. JavaScript can then layer on top to make it richer, faster, and more dynamic for real users.
Practical Principles of Building for Bots
- Ensure core content is in the rendered HTML. If you strip JavaScript from your page entirely, your key text, headings, product descriptions, and structured data should still be there.
- Avoid client-side-only content delivery. Don’t use JavaScript to fetch and inject all your page content from an API after the page loads. Server-side render it.
- Be cautious with infinite scroll and lazy loading. If content only appears when a user scrolls, or when they click a “load more” button, bots won’t see it. Use pagination with real URLs, or ensure the full content is in the initial HTML.
- Keep your DOM clean. Deeply nested, complex JS framework structures add noise to LLM parsing. Semantic HTML, such as proper use of <article>, <section>, <h1> through <h6>, <p>, helps AI systems understand what’s content and what’s chrome.
- Use SSR or SSG for content-critical pages. You don’t need to abandon JavaScript frameworks. Next.js, Nuxt and others all support server-side rendering. Use it where it counts.
Rendering in an AI Search World: The New Reality
The shift from keyword-based search to AI-generated answers is not a future trend; it’s happening now. And the technical implications for how you build and render your website are real.
Three things are true in this new search reality:
AI systems rely on clean, accessible HTML.
If your content isn’t in the raw HTML, it’s invisible to the scrapers feeding LLM knowledge bases.
LLMs have less tolerance for JavaScript dependency.
Google has built a sophisticated deferred rendering pipeline. LLM scrapers largely haven’t. They will time out, give up, and move on.
A clean DOM results in better retrieval.
Well-structured, semantic, server-rendered HTML gives LLM systems the full context they need to vectorise your content accurately — and to surface it when users ask relevant questions.
The equation is simple:
Key Takeaways
- If it's not in the HTML, the content is at risk. Any content that depends on JavaScript execution to appear is invisible to basic scrapers and at risk even with more sophisticated crawlers.
- Rendering issues lead to ranking and retrieval issues. Poor rendering doesn't just affect your Google rankings — it affects whether you appear in AI-generated answers at all.
- JavaScript is not the problem — how you use it is. JavaScript is a powerful tool. Use it to enhance the experience for real users. Don't use it to deliver the core content that bots need to understand your site.
