JavaScript Links Can Hide Content From AI Crawlers, a 41-Day Test Shows
JavaScript-injected navigation can create a material discovery gap between search crawlers and AI-focused bots. In a 41-day field experiment, pages linked only through JavaScript were not discovered by GPTBot, ClaudeBot, PerplexityBot, or other AI crawlers observed in the test. Googlebot could follow JavaScript-generated links, but its coverage declined at greater crawl depths. The experiment,…
A 41-day experiment reveals that JavaScript-injected navigation can prevent certain pages from being discovered by AI crawlers. In the test, pages linked exclusively through JavaScript were not indexed by GPTBot, ClaudeBot, PerplexityBot, or other AI crawlers, while Googlebot could follow these links but with limited coverage at deeper levels.
The experiment, conducted by SEO engineer Vinicius Stanula, highlights the importance of standard HTML anchors for ensuring content accessibility to both conventional search engines and AI-focused crawlers. The test site comprised about 1,000 pages across 21 categories, with half using JavaScript-only links and half using standard HTML links.
AI crawlers failed to discover the JavaScript-only pages, while Googlebot and other Google bots were able to reach them but with reduced depth coverage. Once links were made available in plain HTML, AI crawlers quickly discovered additional pages. This finding underscores the need for enterprise teams to consider crawlability alongside SEO, organic search visibility, and AI assistant discovery when managing content accessibility for various users, including AI systems.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
