How AI crawlers read your robots.txt (and why your * rules may not apply)

Most robots.txt files were initially designed for Googlebot and Bingbot. As AI companies have introduced new crawlers, site owners may unknowingly block AI search or allow training crawlers through their robots.txt files. This article discusses three issues that commonly cause confusion: which AI bots exist, how a crawler determines the rules to follow, and why a well-intended User-agent: * block may be ineffective.

AI user agents can be categorized by their purpose, rather than the company that created them. Answer engines or AI search crawlers fetch pages to be cited in answers, such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot. On-request fetchers, like ChatGPT-User, load a page based on a user's query. Training crawlers collect content to train AI models, including GPTBot, ClaudeBot, and CCBot (Common Crawl).

Site owners often want to allow the first two types of crawlers while blocking the third. However, it is crucial to correctly name the bots in the robots.txt rules, as vendors may change how their bots behave over time. The robots.txt standard, as defined in RFC 9309 and documented by Google, states that the crawler looks for the group with the most specific User-agent line matching its own name.

If found, the crawler follows only that group. If no specific group is found, the crawler follows the * group as a fallback. An important point to note is that the longest matching path wins inside the group, and if an Allow and Disallow rule have the same length, the Allow rule takes precedence. A common mistake occurs when a * group is present in the file, but a specific group for an AI bot is also included.

In such cases, the AI bot ignores the * group entirely. For example, a site owner may block /admin/ and /cart/ paths using a * group, but if OAI-SearchBot has its own group, it will not be affected by these rules. To address this issue, it is necessary to repeat the shared rules inside every named group, ensuring that the AI bots follow the intended restrictions.

This article suggests using a free online checker to test robots.txt configurations for 22 AI crawlers, including live sites or draft files. The checker also scans for an llms.txt file and generates a block like the one above, preserving the existing * rules. By using this tool, site owners can ensure their robots.txt files are correctly configured for both human and AI crawlers.