How a new standard could make it easier for publishers to opt out of AI training

A new standard aimed at giving publishers the ability to collectively opt out of AI training is being proposed by Kevin Kelley, a technical lead for standards at Anthropic. The proposed "policy token" is intended to be a shared signal that can be added to a website's robots.txt file, which currently governs which web crawlers can access and scrape a site. This token would allow publishers to indicate their preferences across multiple AI companies, rather than needing to do so separately for each provider.

The push for this standard comes as publishers have increasingly challenged the use of their content for AI training without permission or compensation. The issue has escalated with the rise of generative AI, as AI crawlers scrape publishers' sites to collect data for model training. While existing standards like robots.txt have been used to signal preferences, they may have limitations and need to be configured for different AI providers.

Several attempts have been made to combat unauthorized scraping, including measures taken by web infrastructure companies like Cloudflare and standards like Really Simple Licensing (RSL), which allows publishers to charge crawlers that visit their sites. However, the proposed policy token differs in that it has garnered support from major AI model developers, including OpenAI, Google, Microsoft, and Anthropic. These companies have stated that they will restrict their AI crawlers from accessing sites with the policy token.

Anthropic has already adopted the signal, and OpenAI plans to implement it in the coming months. Google and Microsoft are expected to support the standard next year. However, Meta, Facebook and Instagram's parent company, is a potential holdout, although it has not yet specified when it will start honoring the policy token. Meta's communications head, Andy Stone, acknowledged the importance of a common signal for conveying preferences to multiple providers and expressed willingness to work with other AI companies on the standard's implementation.