{
  "id": 17370161,
  "title": "AI giants promise publishers new way to block some scraping",
  "url": "https://urgent.news/2026/10/31/ai-giants-promise-publishers-new-way-to-block-some-scraping-17370161",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-31T08:17:27.000Z",
  "source": {
    "name": "The Hindu - Sci-Tech",
    "slug": "the-hindu-sci-tech",
    "url": "https://www.thehindu.com/sci-tech/technology/ai-giants-promise-publishers-new-way-to-block-some-scraping/article71648159.ece"
  },
  "original_language": "en",
  "account": "OpenAI, Google, Microsoft and Anthropic have agreed on Friday to offer publishers a unified method to prevent their content from being scraped for artificial intelligence model training. Publishers typically require multiple blocking commands for each web crawler, but the proposed unified protocol would allow them to implement a single standardized instruction recognized by all AI companies that adopt it. This solution will not be fully operational for several months, with Microsoft initiating the process on February 1, 2027, and Google doing so in 2027. OpenAI plans to roll out the solution in the coming months, while Anthropic has already begun implementation. The announcement comes amid an increase in web crawlers stealing articles to train AI models, leading some publishers to take legal action. The New York Times sued OpenAI and Microsoft in December 2023, alleging unauthorized use of their articles to train ChatGPT. Meanwhile, internet infrastructure provider Cloudflare has blocked AI bots by default on new customer websites since July 2025, and Really Simple Licensing allows publishers to charge bots visiting their pages. The new setting will only affect future model training and will not stop bots from crawling web pages or prevent content scraping unless a publisher specifically disables it.",
  "summary": "Publishers currently have to implement multiple blocking commands for every bot that crawls the web to collect data",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 2,
    "also_reported_by": [
      {
        "outlet": "Channel News Asia",
        "title": "AI giants promise publishers new way to block some scraping",
        "url": "https://urgent.news/2026/10/31/ai-giants-promise-publishers-new-way-to-block-some-scraping",
        "published": "2026-10-31T05:47:00.000Z"
      }
    ]
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}