Most AI Crawlers Do Not Run Your JavaScript
Every GEO article tells you to write helpful content and add schema. Fine advice. It also quietly assumes the machines can read your page, and for a large number of sites that assumption is wrong. AI search runs in two steps. First the engine crawls and stores what it can read. Then, when somebody asks a question, it pulls the best passages and assembles an answer. The failure happens in step…
The article discusses a critical problem with AI crawlers and how they often fail to execute JavaScript on websites, making it impossible for them to read dynamic content generated by scripts. This is a significant issue because many AI systems rely on this step to gather information from web pages.
The study conducted by Vercel in 2024 found that major AI crawlers, including GPTBot, ClaudeBot, and PerplexityBot, only partially or not at all executed JavaScript files. In contrast, Googlebot, which runs through Google's renderer, does execute JavaScript, albeit with a delay. This means that any content built dynamically in the browser, such as product specifications, FAQs, or review counts, is invisible to these crawlers.
To check if your site's content is accessible to these crawlers, you can open the page and compare the raw HTML source with what you see in your browser. If the content appears in the source but not in the live DOM, it is likely not being read by the crawlers. Another method is to use the command `curl -s https://yoursite.com/page | grep -i "a sentence from your page"` to see if the sentence you're looking for appears in the raw HTML.
The solution to this problem is to ensure that all content is available in the raw HTML response sent by the server. This can be achieved through server-side rendering, static generation, or pre-rendering techniques, depending on your site's technology stack. Additionally, it's essential to open the crawlers in your site's robots.txt file to ensure they can access your content.
The article also provides some tips on optimizing your site for AI visibility. Writing in blocks with self-contained sections, leading with concise answers, and adding evidence such as statistics can improve visibility. Using schema markup, such as Article and Product schema, and providing honest last-updated dates can also help. Entity signals, like having your organization's sameAs link to verified profiles, can further boost your site's AI visibility.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
