Google, Anthropic and others to support new AI crawling opt-out standard

Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC have announced their plans to support a new website crawling standard designed to help website operators block artificial intelligence developers from using their content to train AI models. The technology would allow publishers to use a text file titled "robots.txt" to prevent scrapers from accessing their content, either entirely or for specific webpages.

Currently, website operators must make multiple edits to their robots.txt file to block training-related scraping by individual AI providers, which can be cumbersome. The new standard aims to simplify this process by allowing operators to block all scrapers with a single entry. The structure of robots.txt files follows a standardized format called the Robots Exclusion Protocol, which Google, Microsoft, Anthropic and OpenAI plan to incorporate into their proposal.

Microsoft has also introduced a separate setting called AI Grounding, which would prevent its services from sending scraping requests that pertain to answering user questions instead of collecting training data. OpenAI intends to provide similar controls for its users. While Meta Platforms Inc. supports the idea behind the technology, it has not yet committed to implementing it.

The standard is expected to be adopted by Google next year and by Microsoft and OpenAI in the coming months.