Cloudflare Separates AI Training From Search, but Googlebot Rules Need Careful Review

Cloudflare has revamped its AI bot blocking strategy, introducing distinct settings for Search, Agent, and Training traffic. While this provides website owners with finer-grained control over how AI systems interact with their content, it also presents an operational challenge: crawlers used for both search and AI training may be restricted by Training rules. This includes Googlebot when Cloudflare's combined-crawler behavior is in effect.

On September 15, 2026, Cloudflare rolled out the revised system, positioning it as a means for publishers and site operators to allow search discovery without automatically permitting AI training. For businesses reliant on organic search traffic, this means blocking AI training is no longer a straightforward, isolated decision. Website owners must now examine whether their chosen settings also impact multi-purpose crawlers necessary for search indexing.

Cloudflare's new model separates crawler access into three categories:

1. Search, which covers crawlers used for content discovery and indexing, and is allowed by default for new domains.

2. Agent, which covers AI agents accessing content for tasks or retrieval, and is blocked by default on pages that display ads.

3. Training, which pertains to crawlers collecting content for AI model training, and is blocked by default on pages that display ads.

Multi-purpose crawlers, including Googlebot, can be affected when Training is blocked since the combined Search and Training behavior may be restricted. To navigate this complexity, website owners should review their Cloudflare settings, determine which forms of access (search indexing, AI agent visits, AI training, or a combination) are most crucial, and verify that the intended crawler access works post-configuration change, especially for pages containing advertising.

This nuanced approach allows site owners to maintain desirable search visibility while controlling AI training access.