Your Rate Limiter Is a Stampede at the Window Boundary

The fixed window rate limiter lets through double the traffic at the boundary, as the distributed counter is an agreement issue. The token bucket offers a more honest solution. The page was accessed at 00:00:37 on a Tuesday, but the database was struggling and the rate limiter's dashboard appeared fine. The limit was set to 5,000 requests per minute, with the busiest minute recording 4,800 requests.

The limiter was telling the truth, but the database was still failing. Around 23:59:59, two thousand requests landed in the last second of one window, and at 00:00:01, another two thousand requests arrived in the first second of the next window. Despite each window reporting under the limit, the database received four thousand eight hundred requests in roughly two seconds, highlighting the stampede.

This is not a bug in the limiter, but the design working as specified. Two common defaults can fail silently under load: no limiter at all or a fixed-window counter. The fixed-window counter, while simple and cheap, fails because the limit is enforced per window but experienced time as a continuum. The third option is to focus on bursts rather than averages, choosing the algorithm based on the worst second you can tolerate.

The three key questions to consider are: what guarantees does each algorithm provide at the boundary, how does it break when counters need to live on multiple machines, and what should clients do when no requests are allowed.