JavaScript Navigation Can Hide Pages From AI Crawlers, 41-Day Test Finds
JavaScript-powered navigation can make sections of a website difficult, or impossible, for AI crawlers to find. A controlled 41-day experiment reported by Search Engine Land found that GPTBot and ClaudeBot discovered no hierarchy pages available only through JavaScript-injected links, while standard HTML navigation was crawled far more reliably. The finding matters because internal links are not…
JavaScript navigation can hinder search engines and AI crawlers from discovering certain sections of a website. A 41-day experiment demonstrated that GPTBot and ClaudeBot were unable to locate hierarchy pages accessible only through JavaScript-injected links, while standard HTML navigation was crawled more reliably. The experiment involved dividing 1,062 hierarchy pages into two groups - 748 pages with ordinary HTML links and 293 pages with JavaScript-injected links.
Of the HTML-linked pages, only Googlebot discovered 35 of the 348 JavaScript-hierarchy pages during the initial crawl. The study highlights the importance of providing crawlable links for search engines and AI systems, as well as the need for website owners to prioritize crawlability over aesthetics in navigation design.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
