I Argued With an Interviewer About Global Rate Limiting. Neither of Us Were Right

The source discusses the concept of global rate limiting, a technique used to control the amount of requests a system can handle from different regions. The key points are:

1. Global rate limiting has a conservation law: the product of the coordination rate and the worst-case overshoot is equal to the arrival rate multiplied by the number of regions. This means that the design choice of coordination rate and sync frequency is crucial.

2. The initial proposal was for per-request broadcast rate limiting, where each region enforces the limit locally and communicates with other regions in real-time. This approach is found to be extremely inefficient, leading to a massive increase in message traffic and cross-region round trips.

3. The interviewer suggested a broadcast mechanism to share rate limit counts between regions. The reporter argues against this due to the over-engineered nature of the solution and the fact that most rate limiting is approximate rather than requiring distributed consensus.

4. The reporter then explains that the more effective approach is periodic aggregate sync, where each region keeps local counters and synchronizes them every 100 milliseconds or so. This method significantly reduces the message traffic, scaling with the number of active accounts and the sync period rather than the overall request rate.

5. An example is provided to demonstrate the difference between the two approaches. Using five regions and a limit of 1,000 requests per second, per-request broadcast would result in 4,000,000 messages per second, while periodic aggregate sync would only need 200 messages per second.

6. The reporter emphasizes that a static per-region limiter can be ineffective under traffic skew, as it under-allocates the limit to regions with concentrated traffic. This leads to a systematic under-delivery of requests, violating the promised service level.

7. Finally, the author concludes that the conservation law is a fundamental principle to consider when designing rate limiting systems. Once you understand this, most architectural debates about rate limiting become unnecessary.