Our store's robots.txt says no to AI training and yes to AI search. Here's what that actually means
I opened our own robots.txt this week expecting a few lines. It's mostly a list of bots we turned away, and I didn't write any of it by hand. The store runs behind Cloudflare, and Cloudflare's managed robots.txt is switched on. It prepends a block of rules and keeps it updated as new crawlers show up. That's convenient, but "convenient and invisible" is how you end up with a file you can't…
The store's robots.txt file explicitly allows AI search while disallowing AI training. This means the file does not cover AI-generated summaries, content feeding into models, or AI-generated answers, but it does permit building a search index and showing links. The file also lists a number of known bots that are blocked, including GPTBot and ClaudeBot, while allowing Googlebot and other search crawlers.
The store's file also leaves the ai-input signal blank, meaning it is neither granted nor restricted, and is therefore allowed by default. This file is advisory, and if a specific bot is causing issues, a firewall or bot-management rule should be used instead. The file only controls fetching, and a store's product data may still be accessed by AI systems through other means, such as Merchant Center feeds or sitemaps.
The primary goal of the file is to ensure that bots sending people back to the store can read the pages clearly, while bots that do not can't.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
