AI giants promise publishers new way to block some scraping
Publishers currently have to implement multiple blocking commands for every bot that crawls the web to collect data, but the new proposed solution will allow them to enter a single, standardised instruction that's recognised and honoured.
San Francisco - Tech giants OpenAI, Google, Microsoft, and Anthropic have pledged to provide a single method for media outlets and websites to prevent their content from being scraped to train artificial intelligence models. Publishers currently need to implement multiple blocking commands for each web crawler, but the proposed standard will allow them to enter one unified instruction recognized by all AI companies adopting the protocol.
The solution will not be fully implemented for several months, with Microsoft starting in February 2027, Google sometime in 2027, OpenAI in the coming months, and Anthropic already beginning roll-out. The announcement comes amid increased competition from bots that scrape articles to feed AI models, with some publishers like the New York Times taking legal action.
Microsoft and Google stated opting out of AI training may reduce the likelihood of a publisher's content appearing in AI apps and experiences. Meta also expressed support for the effort.
Written by urgent.news from Channel News Asia's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
