{
  "id": 9434587,
  "title": "Cloudflare Separates AI Training From Search Indexing With New Crawler Controls",
  "url": "https://urgent.news/2026/09/23/cloudflare-separates-ai-training-from-search-indexing-with-new",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-23T22:45:30.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/alifar/cloudflare-separates-ai-training-from-search-indexing-with-new-crawler-controls-95"
  },
  "original_language": "en",
  "account": "Cloudflare has introduced new controls to separate AI training from search indexing, addressing the challenge faced by website owners. The new Disallow AI Training directive is published in a site's robots.txt, enabling content to remain discoverable in search results while preventing its use for AI model training. This change is part of Cloudflare's September 15, 2026 update on accountable mixed-use crawlers. Instead of a single Block AI Bots option, Cloudflare now offers distinct controls for Search, Training, and Agent activity. The Disallow AI Training setting communicates a site's preference against AI training, while Search indexing can still be allowed. Additionally, Cloudflare is replacing the Managed Robots.txt with Bot Preference Sync, which automatically aligns a zone's robots.txt with its AI bot policies. This synchronization ensures that a site's stated crawling preferences match its Cloudflare configuration. The new controls allow site owners to have more granular control over their content, enabling search visibility while preventing AI training, which can be particularly useful for business assets like guides, product data, and support content.",
  "summary": "Cloudflare has introduced new AI-crawler controls designed to solve a growing problem for website owners: allowing content to remain discoverable in search results without also permitting that content to be used for AI model training. The central addition, Disallow AI Training , publishes a no-training directive in a site's robots.txt while allowing eligible mixed-use crawlers to continue search…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}