Our System Crashed at 14:22: It Wasn't the Database
Our System Crashed at 14:22: It Wasn't the Database TL;DR: During peak concurrent load, our 5-minute cache unexpectedly evicted keys early due to lock contention. The solution was migrating to asynchronous stale-while-revalidate with an in-memory semaphore. The diff took 18 lines and cut our p99 latency from 1.8s down to 240ms. What Broke in Production Last Tuesday, our primary streaming endpoint…
On a Tuesday, our primary streaming endpoint began experiencing sporadic timeouts. This resulted in a significant increase in latency, jumping from 280ms to 3,400ms. Additionally, the 504 Error Rate surged to 4.2% within an 18-minute period, and the Connection Pool reached full saturation.
Our first thought was that the issue was due to the upstream LLM provider throttling. However, this proved to be incorrect. Despite restarting Celery workers, the problem persisted within just 90 seconds.
The true root cause was an internal "thundering herd" problem. When 300 concurrent requests hit an expired cache key at the same second, every worker simultaneously initiated the same upstream recomputation query. This is demonstrated by the following sequence:
[Request A] ──┐
[Request B] ──┼─► [Expired Cache Key] ──► 300 simultaneous upstream calls
[Request C] ──┘
To resolve this issue, we implemented a non-blocking approach that serves stale data while a single detached coroutine refreshes the cache in the background. The code solution involves using asynchronous lock acquisition to serve stale data while a background task revalidates the cache. The key components are:
1. `get_with_revalidation`: This async function retrieves data from the cache, checking if it's expired. If so, and if the lock for the key is not held, it creates a task to revalidate the cache in the background.
2. `revalidate_background`: This async function acquires the lock for the key and retrieves fresh data from the factory coroutine, then sets the updated cache entry with a specified TTL.
It's important to note that this approach may not be suitable for systems handling strict financial balances or ledger transactions, where stale reads could lead to incorrect results. However, for serving model metadata and prompt routing rules, where stale data is acceptable, this solution effectively addresses the thundering herd problem and reduces latency from 1.8 seconds to 240 milliseconds.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
