Your robots.txt probably does not block the AI crawler you think it blocks
Here is a rule that looks like it blocks OpenAI's crawler: User -agent: * Disallow: /checkout/ Disallow: /customer/ User -agent: GPTBot Crawl-delay: 10 It blocks nothing. GPTBot reads its own group, finds one Crawl-delay line and no Disallow , and crawls everything — including /checkout/ and /customer/ . Under RFC 9309 , a crawler that matches a specific user-agent group uses only that group. It…
The report reveals that many websites' robots.txt files may not effectively block OpenAI's AI crawler, despite the presence of specific rules. The crawler, identified as GPTBot, appears to bypass these restrictions. This discrepancy arises because OpenAI's crawler operates under its own user-agent group, which overrides the wildcard rules typically set by the User-agent: * line.
The author, after cataloging various projects that help ecommerce stores interact with AI systems, found that 11 of the 42 projects focus on crawler policy, while the remaining entries are distributed across different areas such as specifications, structured data, and crawler policy. The author suggests that the distribution is disproportionate, with the crawler policy layer receiving significant attention while other crucial aspects are given less focus.
The report also highlights that OpenAI has shifted its focus away from in-chat Instant Checkout, moving more towards product discovery and feeds. The author advises website owners to start by measuring their current situation, including how often AI systems cite their pages, and then to ensure their crawler policy aligns with RFC 9309 before implementing other measures.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
