{
  "id": 7502411,
  "title": "The rules of the web change today as Cloudflare cracks down on AI bots",
  "url": "https://urgent.news/2026/09/15/the-rules-of-the-web-change-today-as-cloudflare-cracks-down-on-ai-bots",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-15T06:49:06.000Z",
  "source": {
    "name": "Gulf News",
    "slug": "gulf-news",
    "url": "https://gulfnews.com/technology/the-rules-of-the-web-change-today-as-cloudflare-cracks-down-on-ai-bots-1.500674972"
  },
  "original_language": "en",
  "account": "Starting September 15, Cloudflare is implementing new default settings to give website owners more control over how artificial intelligence companies use their content. Domains joining Cloudflare will automatically block AI bots used for training or as agents on pages displaying ads, while search crawlers will remain allowed by default. Existing free-tier Cloudflare customers will also be affected if they haven't adjusted their AI bot settings by the deadline.\n\nThe move could impact publishers, creators, and AI companies as the debate over content access and usage escalates. Cloudflare separates AI crawlers into three categories: search, training, and agent. Website owners can choose to allow, block, or block specific pages for each category. This distinction is crucial because publishers may want their articles to appear in search results but prevent them from being collected for AI training without compensation.\n\nCloudflare announced the policy in July but the new defaults will take effect on September 15. For new customers and sites, search crawling will be allowed by default, while training and agent activities are blocked on ad pages. Existing free customers will also be subject to the new defaults unless they change their preferences. Crawlers performing multiple functions will be subject to the most restrictive setting, potentially blocking them if they're used for both search and training.\n\nThe change is particularly significant for publishers who are concerned about the balance between allowing AI companies to use their content and the traffic generated by AI systems. Cloudflare also offers analytics to some customers showing how frequently AI operators crawl their sites compared to the referral traffic they generate. This move aims to help website owners distinguish between AI services that help users discover content and those that consume it for other purposes.",
  "summary": "The rules governing how artificial intelligence companies crawl the web are changing from today, as Cloudflare begins enforcing new defaults designed to give website owners greater control over how their content is used. From September 15, new domains joining Cloudflare will automatically block bots classified as being used for AI training or AI agents on pages displaying advertisements, while…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}