The biggest names in AI are backing a new way to opt out of scraping

The leading figures in artificial intelligence have announced their commitment to a new standard that would enable publishers to opt out of being scraped by AI for training purposes. OpenAI, Google, Microsoft, and Anthropic have all expressed their support for the new control, which would allow publishers to apply their AI training preferences across multiple crawlers simultaneously.

This would simplify the process of managing AI crawlers, as publishers would no longer need to signal their preferences individually to each AI company. The proposed standard, known as a shared "policy token" signal, will be integrated into a website's robots.txt file, which is currently used to dictate the behavior of web crawlers.

Kevin Kelley, a technical lead for standards at Anthropic, has been instrumental in developing this proposal, which lives within the Internet Engineering Task Force (IETF). By adopting this signal, publishers can prevent AI crawlers from accessing their sites, thereby reducing the amount of content that is scraped and used to train AI models.

Currently, there are alternative measures in place to combat unauthorized scraping, such as the Really Simple Licensing (RSL) standard and Cloudflare's default blocking of AI crawlers. Microsoft, for instance, will provide more granular controls, granting publishers the ability to opt out of AI training or AI grounding, which is the process of grounding an AI model in verified sources to minimize errors and misinformation.

However, it is important to note that websites will still be visible in Bing's search results, regardless of the preferences set by publishers. While Google and Microsoft intend to support the new standard next year, OpenAI plans to implement it within the coming months.