Cloudflare: Stay discoverable in search while disallowing AI training

Cloudflare has introduced a new feature to help website owners remain discoverable in search while preventing AI training. This is due to the tradeoff caused by mixed-use crawlers that can serve both search and AI training purposes. If a site owner refuses to allow one, they inadvertently block the other. Cloudflare's new Disallow AI Training setting allows site owners to maintain search indexing while blocking AI training.

Major search engines such as Apple, Google, and Microsoft have committed to respecting this setting. The new setting will complement existing options for blocking training crawlers and ads. It provides site owners with granular control over how their content is used without sacrificing visibility in search results.