{
  "id": 18983890,
  "title": "Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping",
  "url": "https://urgent.news/2026/11/08/google-and-microsoft-among-those-signing-up-to-openais-plan-for-a-new",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-08T17:05:00.000Z",
  "source": {
    "name": "TechRadar",
    "slug": "techradar",
    "url": "https://www.techradar.com/pro/google-and-microsoft-among-those-signing-up-to-openais-plan-for-a-new-website-crawling-standard-including-opting-out-of-internet-scraping"
  },
  "original_language": "en",
  "account": "Google, Microsoft, OpenAI, and Anthropic have all pledged to respect a single robots.txt rule that acts as a cross-company opt-out from AI training. This new line in the robots.txt file tells AI developers whether a website's content can be used to train their models, instead of each company having to write separate rules for their own crawlers. While this is seen as progress for webmasters, the actual scope of the commitment is narrower than often reported. The rule indicates a preference for how content may be used, not an outright ban on bots from accessing it. It does not address content that has already been scraped, and not all major players have fully implemented the signal yet. OpenAI and Anthropic have started to implement the rule, Microsoft will begin doing so on February 1, 2027, and Google aims to do so in 2027. However, each company approaches the implementation differently, and Meta has not fully committed to the rule. The rule's effectiveness is limited, as it does not stop the bots from crawling the content, and it relies on the cooperation of the participating companies.",
  "summary": "One robots.txt line could soon tell Google, Microsoft, OpenAI, and Anthropic not to train on your site, but it won't stop their bots from crawling it.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}