Your zero-downtime deploy is probably fine. Check your p99 before you believe it.
I went looking for dropped requests during a rolling restart and found something more annoying than dropped requests: a deploy that looks perfect and isn't. Setup is deliberately boring. Two Node/Express replicas behind nginx, ten clients hammering an endpoint that takes three seconds, docker stop on one replica halfway through. The app is the version most of us have shipped at some point, with…
The article discusses a scenario where a zero-downtime deploy appears successful but actually fails during the process. The setup includes two Node/Express replicas behind Nginx, with a ten-client load test on an endpoint that takes three seconds to complete. During a Docker stop on one replica, the app's lack of signal handling results in mid-flight requests dying.
Unexpectedly, Nginx detects the upstream connection drop and retries the request, causing clients to experience twice the latency. While error rate metrics may appear flawless, latency metrics reveal a spike during each deploy, which may be overlooked. The author explains that once headers are in transit, Nginx cannot prevent the retry.
The solution involves removing the instance from the load balancer before sending SIGTERM, ensuring no requests are sent to a non-responsive instance. This approach eliminates the issue, as demonstrated by a zero-failure rate during testing. The article emphasizes the importance of monitoring latency rather than just error rates to detect potential issues during deploys.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.