We let crawlers fetch the pages we never want indexed, because a Disallowed URL never reads its noindex

Munchable is a barcode scanner for digestive conditions, with 426 indexable URLs on its marketing site. The file that determines which of these URLs a crawler may fetch is 29 lines of TypeScript code. The only significant line in the code is "disallow : [/api/, /auth/]", which disallows crawling of just two paths out of the total indexable URLs.

This goes against the conventional understanding, as both "disallow" and "noindex" are typically used to prevent a page from being indexed. However, the article explains that when both directives are applied, the crawler never fetches the page, thus never seeing the "noindex" directive. Consequently, the URL remains indexed as a bare result, even if it should not be.

The issue is particularly relevant for pages that users might consider private, such as checkout returns, authentication pages, or any page a signed-in user accesses after a redirect. The article provides an example of a homepage link in the pricing section, which could be discovered without being fetched due to the "noindex" header.

Munchable's robots.ts file only disallows the two restricted paths, allowing all other pages to be indexed and including a "noindex" directive within the response body. The article emphasizes the importance of understanding the difference between the "disallow" and "noindex" directives and how combining them can unintentionally leave certain pages indexed.