{
  "id": 19677974,
  "title": "Our store's robots.txt says no to AI training and yes to AI search. Here's what that actually means",
  "url": "https://urgent.news/2026/11/12/our-stores-robots-txt-says-no-to-ai-training-and-yes-to-ai-search",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-12T17:02:24.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/elias_0549e0e2ef8cf6ad89a/our-stores-robotstxt-says-no-to-ai-training-and-yes-to-ai-search-heres-what-that-actually-means-324p"
  },
  "original_language": "en",
  "account": "The store's robots.txt file explicitly allows AI search while disallowing AI training. This means the file does not cover AI-generated summaries, content feeding into models, or AI-generated answers, but it does permit building a search index and showing links. The file also lists a number of known bots that are blocked, including GPTBot and ClaudeBot, while allowing Googlebot and other search crawlers. The store's file also leaves the ai-input signal blank, meaning it is neither granted nor restricted, and is therefore allowed by default. This file is advisory, and if a specific bot is causing issues, a firewall or bot-management rule should be used instead. The file only controls fetching, and a store's product data may still be accessed by AI systems through other means, such as Merchant Center feeds or sitemaps. The primary goal of the file is to ensure that bots sending people back to the store can read the pages clearly, while bots that do not can't.",
  "summary": "I opened our own robots.txt this week expecting a few lines. It's mostly a list of bots we turned away, and I didn't write any of it by hand. The store runs behind Cloudflare, and Cloudflare's managed robots.txt is switched on. It prepends a block of rules and keeps it updated as new crawlers show up. That's convenient, but \"convenient and invisible\" is how you end up with a file you can't…",
  "key_points": [
    "Store's robots.txt allows AI search but disallows AI training.",
    "File permits building search index and showing links, but blocks GPTBot and ClaudeBot.",
    "File is advisory; specific bot issues require firewall or bot-management rules."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}