Cloudflare Blocks AI Agents From Ad-Supported Pages
Cloudflare has started blocking artificial intelligence agents by default on pages that carry ads, putting the tools consumers use to compare products in direct conflict with the sites those tools need to read. The new preset for ad-supported domains allows search crawlers, refuses AI training and blocks user-directed agents on any page that shows an […] The post Cloudflare Blocks AI Agents From…
Cloudflare has begun blocking AI agents by default on pages that contain advertisements, which creates a conflict between the tools consumers use for product comparisons and the websites those tools rely on. The new default setting allows search crawlers, but it blocks AI training agents and user-directed agents that access any page with an ad.
There is no current option for agents to opt-out of the block. Cloudflare handles a significant portion of web traffic, about a fifth of it, according to TechCrunch. The company's reasoning is that ads indicate a page was intended for human viewing; AI agents that retrieve pages on behalf of shoppers do not see these ads. Cloudflare aims to reduce mixed-use crawler traffic to zero within a year and allows site owners to control content visibility in AI summaries by mid-2027.
While retailers like Amazon, Walmart, and Target are testing their own strategies regarding AI agents, Cloudflare caters to both sides of the issue, recently launching Kitesurf, a cloud-hosted browser designed for AI agents.
Written by urgent.news from PYMNTS's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
