Google joins OpenAI and Anthropic in promising new scraping controls

Google, Anthropic, OpenAI, and Microsoft have agreed to follow a new robots.txt directive. This directive will allow publishers to control how their content is scraped for training generative AI models. The proposal, which has been under consideration at IETF, the internet's open standards organization, was written by Anthropic's technical lead for standards, Kevin Kelley.

Consequently, website owners will only need to set up a new user agent in the robots.txt file for AI training. This user agent will look like "User-Agent: __usage_ai-training". Following this, there will be a "Disallow: /" directive, which will block crawlers specifically designed for AI training from accessing the site. However, the "Allow: /press-kit/" directive offers website operators the option to permit scraping for AI training for specific sections of the page, such as a press kit.

This new user-agent section does not hinder crawling entirely or prevent content from appearing in Google search. For instance, Google will interpret the directive as a stop sign exclusively for the "Google-Extended" user agent, which crawls websites for materials used in training Gemini models.

It's essential to note that this directive cannot reverse any content already scraped by the companies. If they had previously crawled a page, the content remains accessible. Google indicated that it will support the new directive in early 2027, while Anthropic has already started implementing it. Microsoft announced that it will start supporting the directive on February 1, 2027, and OpenAI pledged to implement the new signal in the coming months.