{
  "id": 12414807,
  "title": "0 of 550 top homepages send noai. 21 send Bing's noarchive.",
  "url": "https://urgent.news/2026/10/06/0-of-550-top-homepages-send-noai-21-send-bings-noarchive",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-06T16:06:59.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/mahirhir/0-of-550-top-homepages-send-noai-21-send-bings-noarchive-38p9"
  },
  "original_language": "en",
  "account": "In a study of the top 550 webpages, it was found that none of them send directives related to noai or noimageai, nor do they use nosnippet. Out of the 550, only 21 send either noarchive or nocache, which are signals to prevent pages from appearing in Bing Chat answers or to limit their use in training. This suggests that the primary AI policy for these top websites is primarily managed through the robots.txt file.",
  "summary": "I fetched the homepages of the Tranco top 1,000 and read every meta robots tag and every X-Robots-Tag header they sent. 550 homepages came back as real pages. None of the 550 sends noai or noimageai . None sends nosnippet either, and the 47 that send max-snippet all set it to -1 , which means \"no limit\". The page-level AI rule that big sites actually use is Bing's: 21 of the 550 send noarchive or…",
  "key_points": [
    "None of the top 550 webpages send noai or noimageai directives.",
    "Only 21 send either noarchive or nocache signals.",
    "AI policies likely managed through robots.txt file."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}