After scraping the Internet, AI giants pledge ‘no scraping’ tool for publishers

SAN FRANCISCO, October 31 — In a concerted effort to address the plight of publishers, OpenAI, Google, Microsoft, and Anthropic have announced plans to introduce a unified solution to prevent their content from being scraped for AI model training. This move comes in response to mounting criticism and legal disputes from news outlets.

Publishers must currently implement individual blocking commands for each web crawler, but the proposed protocol will enable a single, standardized instruction to be recognized and respected by any AI company adopting it. The implementation timeline varies: Microsoft will commence the rollout on February 1, 2027, Google in 2027, OpenAI within the coming months, and Anthropic has already begun the process.

The issue has gained traction as publishers face an influx of bots that scrape their articles to fuel AI models. Notable publishers, such as the New York Times, have even taken the matter to court. In December 2023, the Times filed a lawsuit against OpenAI and Microsoft, alleging unauthorized use of millions of articles to train ChatGPT, invoking the fair use copyright exception.

Meanwhile, Cloudflare has blocked AI bots by default for new customer sites since July 2025, and Really Simple Licensing allows publishers to charge bots visiting their pages. However, the new setting will only affect future model training and does not restrict bots from crawling content, which can still be scraped by default unless a publisher explicitly changes the setting, as Microsoft clarified in a blog post.

By offering a common signal to convey content preferences to multiple providers simultaneously, Microsoft and Google believe this initiative is a crucial step forward. Meta's Andy Stone echoed this sentiment, expressing support for the effort, as reported by AFP.