Your app went viral. Adding servers made it worse. Here's why.
It's midnight, someone huge shares your app, 100,000 people show up — and the page dies. You add three servers. It gets worse . The rule that finally made this click for me: before you add capacity, find out where the request is actually waiting. Follow one slow request and the whole "scaling" story falls out in order: It's usually the database , not the app — the app only looks busy holding…
At midnight, a major figure shared your app, and suddenly, 100,000 users flooded in - causing the page to crash. Adding three servers did not resolve the issue. The crucial insight that finally made sense of the situation was to first identify the stage where the request was stuck. The request was usually held up by the database, not the app itself - the app merely appeared to be occupied with unfinished tasks.
Adding more app servers only led to more waiting users, one cook with numerous unfinished dishes. The bottlenecks were found by scrutinizing each step of the request's journey. The database was the frequent culprit, with connection pools exhausted due to multiple requests hitting the same database. To optimize, indexing the problematic query and using caching are recommended.
However, care must be taken when the hot key expires, as it can trigger a stampede. Queuing the slow work helps manage the backlog. Asynchronous processing might seem like a solution, but it can lead to a significant delay - sometimes hours. Retries without proper backoff and jitter can exacerbate outages. Each fix uncovers the next bottleneck, turning failure into a learning opportunity.
A visualization was created to illustrate the request's journey through each fix, revealing where the slow request was waiting. The author is open to hearing similar stories.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
