{
  "id": 7908929,
  "title": "Cloudflare Separates AI Training Controls From Search Indexing for Website Owners",
  "url": "https://urgent.news/2026/09/17/cloudflare-separates-ai-training-controls-from-search-indexing-for",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-17T00:30:30.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/alifar/cloudflare-separates-ai-training-controls-from-search-indexing-for-website-owners-152a"
  },
  "original_language": "en",
  "account": "Cloudflare has introduced a new setting called Disallow AI Training, which allows website owners to control whether their content can be used for AI model training without blocking search indexing. This setting is part of Cloudflare's Bot Management and AI Crawl Control toolbox and distinguishes between crawlers based on their purpose: searching, training models, or acting on behalf of users. The new setting addresses the limitations of the previous Block AI Bots control, which was a one-size-fits-all restriction that could impact search discovery. By publishing a no-training directive through Bot Preference Sync, Cloudflare's Disallow AI Training setting allows accountable mixed-use crawlers to continue indexing a site for search while prohibiting them from using its content for AI training. Cloudflare identifies three types of crawlers: Search (crawling to build a search index), Training (crawling to train or fine-tune AI models), and Agent (human-directed or bot-assisted access). The Disallow AI Training setting applies specifically to the Training category, recognizing that some crawlers, such as those from Amazon, Anthropic, Meta, and OpenAI, are expected to honor the preference while still indexing the site for search. The rollout of this new setting involves deprecating the legacy Block AI Bots control and migrating existing customers to the new granular Search, Training, and Agent options. Website owners will need to review their crawler policies and configure their domains accordingly in Cloudflare's Bot Preference Sync. As AI and search results increasingly overlap, businesses should carefully document their decisions about AI access to ensure content visibility and maintain an effective content strategy.",
  "summary": "Cloudflare has introduced a Disallow AI Training setting intended to solve a difficult choice for website owners: limiting the use of their content for AI model training without blocking search indexing . The control is part of Cloudflare's Bot Management and AI Crawl Control toolbox, and it distinguishes among crawlers based on whether they search, train models, or act on behalf of users. The…",
  "key_points": [
    "Cloudflare introduces \"Disallow AI Training\" setting for website owners.",
    "Differentiates crawlers based on purpose: search, training, or agent.",
    "Allows indexing for search while blocking AI model training."
  ],
  "editors_take": "Website owners can now prevent their content from being used for AI model training without affecting search indexing, thanks to Cloudflare's new Disallow AI Training setting.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}