Urgent.News

What's breaking now, across hundreds of outlets.

AI

Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping

One robots.txt line could soon tell Google, Microsoft, OpenAI, and Anthropic not to train on your site, but it won't stop their bots from crawling it.

Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping

Google, Microsoft, OpenAI, and Anthropic have all pledged to respect a single robots.txt rule that acts as a cross-company opt-out from AI training. This new line in the robots.txt file tells AI developers whether a website's content can be used to train their models, instead of each company having to write separate rules for their own crawlers.

While this is seen as progress for webmasters, the actual scope of the commitment is narrower than often reported. The rule indicates a preference for how content may be used, not an outright ban on bots from accessing it. It does not address content that has already been scraped, and not all major players have fully implemented the signal yet.

OpenAI and Anthropic have started to implement the rule, Microsoft will begin doing so on February 1, 2027, and Google aims to do so in 2027. However, each company approaches the implementation differently, and Meta has not fully committed to the rule. The rule's effectiveness is limited, as it does not stop the bots from crawling the content, and it relies on the cooperation of the participating companies.

Rewritten by urgent.news; may contain errors — check the original.

Source: techradar.com

Read the original at techradar.com →

Report a problem with this story

More in AI

How to Give an AI Agent a Spending Limit (and the Tools That Do It)

Quick answer: There are four ways to cap what an AI agent can spend: give it a virtual card with network-enforced limits (Stripe Issuing for agents); give it a wallet with per-transaction and…

  • Four main methods to cap AI agent spending: virtual cards, wallets, policy layers, x402
  • Virtual cards through Stripe enforce limits per authorization or time period
  • Wallets like Coinbase Agentic Wallets set per-transaction and per-session caps

GPT-6 Astra Passed 69.9% of Hidden Checks. It Fully Finished 26.4% of Real Edit Tasks

Book: Ship It — the pragmatic startup tech stack Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go My project: Hermes IDE | GitHub — an IDE…

  • GPT-6 Astra achieved 69.9% success in hidden acceptance checks
  • 26.4% of tasks fully passed real edit tasks
  • Co-Design Bench paper submitted to arXiv on October 26

AI Agents Work in Demos. So Why Do They Fail in Production?

I've noticed something interesting about AI agent development. Building an agent that works in a demo is no longer particularly difficult.

  • AI agents excel in demos but face challenges in production environments.
  • Common issues include calling wrong tools, repeating actions, and getting stuck on simple tasks.
  • Designing agents with clear responsibilities and proper API integration reduces production risks.

Claude Meets Copilot Studio

This is the one I've been building toward. Over the last few posts we deployed Claude in Microsoft Foundry, said hello to it from code, and taught it to turn a messy support ticket into clean…

  • Claude model integrated into Microsoft Copilot Studio
  • Admin enables Anthropic models in Microsoft 365 Admin Center
  • Agent created with system prompt and knowledge sources

Write a Claude Code skill in 10 minutes

Written by an AI agent (Claude), posted from its operator's account. Field names and behaviour were checked against the official Claude Code skills docs; I could not run the example on a live install…

  • Create a folder with SKILL.md file
  • Add YAML frontmatter in SKILL.md
  • Specify name, description, arguments

More from Sunday 8 November →