{
  "id": 20516977,
  "title": "~600K bot requests a day hit my origin. A 24-hour Cloudflare cache with purge-on-change fixed it (on the Free plan)",
  "url": "https://urgent.news/2026/11/17/600k-bot-requests-a-day-hit-my-origin-a-24-hour-cloudflare-cache-with",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-17T23:44:47.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/hipa-ai/600k-bot-requests-a-day-hit-my-origin-a-24-hour-cloudflare-cache-with-purge-on-change-fixed-it-2ffd"
  },
  "original_language": "en",
  "account": "In mid-October, clinical-trials search site hipa.ai experienced a surge in bot requests, with approximately 600,000 bot requests per day hitting its origin. The site, built on Next.js and Azure App Service, rendered pages from scratch for each request. About 95% of the events came from bots, with only around 250-370 real visitors per week. The bots included a headless residential-proxy botnet, OAI-SearchBot, and ClaudeBot.\n\nThe bots either bypassed cache rules or sent Cache-Control: private, no-store headers, causing Cloudflare to pass every request through to the origin. This resulted in a significant load on the Next.js origin, leading to performance issues and even a broken build live in production. To address this, a 24-hour Cloudflare cache with purge-on-change was implemented.\n\nThe cache rule, named [BOT-SHIELD], was built using a small TypeScript module. It cached every route that rendered the same for every visitor, such as /condition/*, /study/*, /drug/*, etc. The cache was set to expire after 24 hours, and the rule was designed to purge content when it changed. This approach ensured that the cached content was always fresh, eliminating the need for expensive renders on the origin.\n\nThe implementation on the Free plan was successful, as purging on the Free plan worked without requiring a paid plan. The purge calls reduced the origin's egress meter from 8.8-9.7 GB per day to 1.1-2.5 GB per day, resulting in an 80% drop in origin egress overnight. This significant reduction in traffic not only improved performance but also prevented potential issues that could have led to a broken build being live in production.",
  "summary": "In mid-October I looked at Cloudflare analytics for hipa.ai , the clinical-trials search site I build alone. On a typical day there were about 455,000 requests marked DYNAMIC and another 141,000 marked MISS . Every one of them went through to my Next.js origin on Azure App Service, which rendered the page from scratch against MongoDB and Elasticsearch. Very few of those requests were people. When…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}