{
  "id": 19991842,
  "title": "How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More",
  "url": "https://urgent.news/2026/11/14/how-to-block-ai-crawlers-with-robots-txt-gptbot-claudebot-google",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-14T14:32:22.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/choruswire/how-to-block-ai-crawlers-with-robotstxt-gptbot-claudebot-google-extended-and-more-92c"
  },
  "original_language": "en",
  "account": "Various AI companies each maintain distinct robots.txt agents to control their training data collection processes. These agents differ, such as OpenAI's GPTBot, Anthropic's ClaudeBot, Google's Google-Extended, Apple's Applebot-Extended, and Common Crawl's CCBot. Blocking one agent does not prevent others from crawling your site, as robots.txt is a voluntary request rather than a strict order. Each company has its own rules for how content can be used, but none guarantee complete exclusion from every AI training dataset. Additionally, search bots and user-triggered fetches use different agents, which are not affected by robots.txt controls. While robots.txt can be used to request that certain content not be used for training, it does not guarantee complete removal from any existing AI models.",
  "summary": "Originally published on ChorusWire . Each AI company documents separate robots.txt agents for training, search and user-triggered fetches. Blocking one does not block the others, and robots.txt is voluntary. To communicate that you do not want certain crawlers collecting your site's content for AI training, use the robots.txt controls documented by each provider. These controls differ by company:…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}