{
  "id": 14509604,
  "title": "Cloudflare Separates AI Training From Search, but Googlebot Rules Need Careful Review",
  "url": "https://urgent.news/2026/10/14/cloudflare-separates-ai-training-from-search-but-googlebot-rules-need",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-14T18:30:30.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/alifar/cloudflare-separates-ai-training-from-search-but-googlebot-rules-need-careful-review-4o3e"
  },
  "original_language": "en",
  "account": "Cloudflare has revamped its AI bot blocking strategy, introducing distinct settings for Search, Agent, and Training traffic. While this provides website owners with finer-grained control over how AI systems interact with their content, it also presents an operational challenge: crawlers used for both search and AI training may be restricted by Training rules. This includes Googlebot when Cloudflare's combined-crawler behavior is in effect.\n\nOn September 15, 2026, Cloudflare rolled out the revised system, positioning it as a means for publishers and site operators to allow search discovery without automatically permitting AI training. For businesses reliant on organic search traffic, this means blocking AI training is no longer a straightforward, isolated decision. Website owners must now examine whether their chosen settings also impact multi-purpose crawlers necessary for search indexing.\n\nCloudflare's new model separates crawler access into three categories:\n1. Search, which covers crawlers used for content discovery and indexing, and is allowed by default for new domains.\n2. Agent, which covers AI agents accessing content for tasks or retrieval, and is blocked by default on pages that display ads.\n3. Training, which pertains to crawlers collecting content for AI model training, and is blocked by default on pages that display ads.\n\nMulti-purpose crawlers, including Googlebot, can be affected when Training is blocked since the combined Search and Training behavior may be restricted. To navigate this complexity, website owners should review their Cloudflare settings, determine which forms of access (search indexing, AI agent visits, AI training, or a combination) are most crucial, and verify that the intended crawler access works post-configuration change, especially for pages containing advertising. This nuanced approach allows site owners to maintain desirable search visibility while controlling AI training access.",
  "summary": "Cloudflare has replaced its former one-click AI bot blocking approach with separate controls for Search, Agent and Training traffic . The change gives website owners more specific choices over how AI systems access their content, but it also creates an important operational issue: crawlers that serve both search and AI training purposes can be affected by Training rules. That includes Googlebot…",
  "key_points": [
    "Cloudflare introduces separate settings for Search, Agent, and Training AI traffic.",
    "Googlebot may be restricted by Training rules when combined-crawler behavior is active.",
    "Website owners must review settings to balance search visibility and AI training control."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}