Cloudflare launches tool to block AI training but keep search
Mixed-use crawlers account for 36.6% of verified crawler traffic on Cloudflare’s network.
Cloudflare has introduced a new feature to help website owners remain discoverable in search while preventing AI training. This is due to the tradeoff caused by mixed-use crawlers that can serve both search and AI training purposes. If a site owner refuses to allow one, they inadvertently block the other. Cloudflare's new Disallow AI Training setting allows site owners to maintain search indexing while blocking AI training.
Major search engines such as Apple, Google, and Microsoft have committed to respecting this setting. The new setting will complement existing options for blocking training crawlers and ads. It provides site owners with granular control over how their content is used without sacrificing visibility in search results.
Written by urgent.news from Hacker News's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.
- Cloudflare: Stay discoverable in search while disallowing AI training blog.cloudflare.com
