{
  "id": 17321100,
  "title": "Google, Anthropic and others to support new AI crawling opt-out standard",
  "url": "https://urgent.news/2026/10/31/google-anthropic-and-others-to-support-new-ai-crawling-opt-out",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-31T02:00:23.000Z",
  "source": {
    "name": "SiliconANGLE",
    "slug": "siliconangle",
    "url": "https://siliconangle.com/2026/10/30/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard/"
  },
  "original_language": "en",
  "account": "Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC have announced their plans to support a new website crawling standard designed to help website operators block artificial intelligence developers from using their content to train AI models. The technology would allow publishers to use a text file titled \"robots.txt\" to prevent scrapers from accessing their content, either entirely or for specific webpages. Currently, website operators must make multiple edits to their robots.txt file to block training-related scraping by individual AI providers, which can be cumbersome. The new standard aims to simplify this process by allowing operators to block all scrapers with a single entry. The structure of robots.txt files follows a standardized format called the Robots Exclusion Protocol, which Google, Microsoft, Anthropic and OpenAI plan to incorporate into their proposal. Microsoft has also introduced a separate setting called AI Grounding, which would prevent its services from sending scraping requests that pertain to answering user questions instead of collecting training data. OpenAI intends to provide similar controls for its users. While Meta Platforms Inc. supports the idea behind the technology, it has not yet committed to implementing it. The standard is expected to be adopted by Google next year and by Microsoft and OpenAI in the coming months.",
  "summary": "Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC today announced plans to support an upcoming website crawling standard. The technology will enable website operators to block artificial intelligence developers from using their content to train their models. Many websites contain a text file titled “robots.txt” that informs crawlers what content they may access. […] The post Google,…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 4,
    "also_reported_by": [
      {
        "outlet": "Times of India",
        "title": "Anthropic’s Daniela Amodei questions OpenAI’s Pentagon deal: Was it opportunistic",
        "url": "https://urgent.news/2026/10/30/anthropics-daniela-amodei-questions-openais-pentagon-deal-was-it",
        "published": "2026-10-30T11:40:10.000Z"
      },
      {
        "outlet": "Nikkei Asia",
        "title": "Japan asks Anthropic, OpenAI for more access to advanced models",
        "url": "https://urgent.news/2026/10/30/japan-asks-anthropic-openai-for-more-access-to-advanced-models",
        "published": "2026-10-30T13:55:51.000Z"
      },
      {
        "outlet": "Techmeme",
        "title": "OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI across multiple crawlers (Emma Roth/The Verge)",
        "url": "https://urgent.news/2026/10/30/openai-google-microsoft-and-anthropic-say-they-will-adopt-a-new",
        "published": "2026-10-30T21:05:00.000Z"
      }
    ]
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}