{
  "id": 12825314,
  "title": "I've Been Deindexed by Google",
  "url": "https://urgent.news/2026/10/08/ive-been-deindexed-by-google",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-08T07:59:30.000Z",
  "source": {
    "name": "Lobsters",
    "slug": "lobsters",
    "url": "https://kennyqin.com/deindexed-by-google/"
  },
  "original_language": "en",
  "account": "Earlier this year, I made the decision to prevent my website from being indexed by search engines, particularly due to the growing issue of AI-driven web scraping. While I am aware that as long as my blog remains accessible on the public internet, there are no foolproof methods to stop it, I hoped some level of respect would be granted. Initially, I implemented a broad disallow directive in the robots.txt file, but after several weeks without any change, I resorted to setting the robots meta tag. However, I soon discovered that merely denying the bot access to my website was insufficient, as it needed to actually visit the site to recognize the meta tag instructing it not to index the pages. Consequently, I had to permit the bots responsible for deindexing, including several common ones, to access my website and subsequently remove them from the robots.txt whenever I noticed my website had been deindexed by them. After implementing these changes, it has been over three months since my last modification to achieve Google's deindexing, and today, I am delighted to confirm that my website is now free from Google's indexing.",
  "summary": null,
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}