Cloudflare Helps Website Owners Block AI Scraping Without Losing Search Visibility

Cloudflare, a connectivity cloud company, has introduced a new feature that allows website owners to block crawlers from using their content for AI training while still maintaining search visibility, according to a Sept. 15 press release. This Disallow AI Training setting effectively addresses the issue posed by mixed-use crawlers that collect data for both search indexing and AI training. Prior to this solution, blocking these crawlers from AI training would also restrict their access to search results.

Cloudflare's new setting is designed to comply with the requirements set by major tech giants such as Apple, Google, and Microsoft, or encourages these companies to commit to the same. Additionally, several other companies already operate separate crawlers for search and training, enabling Cloudflare to block training crawlers without impacting the search ones.

In a statement, Cloudflare Co-Founder and CEO Matthew Prince highlighted the importance of preserving the openness of the internet while empowering website owners with control over their work's usage. "This is how we make the internet better," Prince said. "We look forward to continued collaborative engagement with companies like Apple, Google and Microsoft as we work together to build a healthy ecosystem."

Cloudflare's Product Director, Bryan Becker, shared that discussions with crawler operators began as early as July 2023. The response was encouraging, with nearly all operators agreeing that website owners should have control and transparency into how their content is utilized. Becker added that such choices need to be respected to ensure the integrity of the web.

According to a PYMNTS Intelligence report titled "How Enterprises Can Build a 'Know Your Agent' Defense: Digital Identity Verification in the Age of Bots," nearly 90% of organizations face challenges in managing bot activity. As AI-powered agents interact with platforms alongside human users, weak or outdated controls may allow malicious bots to infiltrate while legitimate users are mistakenly blocked. The PYMNTS report emphasizes the need for robust controls to prevent this from happening.