~600K bot requests a day hit my origin. A 24-hour Cloudflare cache with purge-on-change fixed it (on the Free plan)
In mid-October I looked at Cloudflare analytics for hipa.ai , the clinical-trials search site I build alone. On a typical day there were about 455,000 requests marked DYNAMIC and another 141,000 marked MISS . Every one of them went through to my Next.js origin on Azure App Service, which rendered the page from scratch against MongoDB and Elasticsearch. Very few of those requests were people. When…
In mid-October, clinical-trials search site hipa.ai experienced a surge in bot requests, with approximately 600,000 bot requests per day hitting its origin. The site, built on Next.js and Azure App Service, rendered pages from scratch for each request. About 95% of the events came from bots, with only around 250-370 real visitors per week. The bots included a headless residential-proxy botnet, OAI-SearchBot, and ClaudeBot.
The bots either bypassed cache rules or sent Cache-Control: private, no-store headers, causing Cloudflare to pass every request through to the origin. This resulted in a significant load on the Next.js origin, leading to performance issues and even a broken build live in production. To address this, a 24-hour Cloudflare cache with purge-on-change was implemented.
The cache rule, named [BOT-SHIELD], was built using a small TypeScript module. It cached every route that rendered the same for every visitor, such as /condition/*, /study/*, /drug/*, etc. The cache was set to expire after 24 hours, and the rule was designed to purge content when it changed. This approach ensured that the cached content was always fresh, eliminating the need for expensive renders on the origin.
The implementation on the Free plan was successful, as purging on the Free plan worked without requiring a paid plan. The purge calls reduced the origin's egress meter from 8.8-9.7 GB per day to 1.1-2.5 GB per day, resulting in an 80% drop in origin egress overnight. This significant reduction in traffic not only improved performance but also prevented potential issues that could have led to a broken build being live in production.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.