How a 30-Second Pod Grace Period Lost the Race Against a 300-Second ALB Deregistration Delay
14:32 UTC. PagerDuty fires on the checkout service's error-rate alarm. Datadog shows the 502 rate on POST /checkout jumping from a baseline of 0.02% to 4.8%, right in the middle of a rolling deploy that had shipped a dozen times before without anyone noticing it. By 14:36 the deploy finishes, the 502s stop, and the dashboard looks normal again. Nobody asked for this one to rotate twelve pods for…
A 30-second grace period for deregistering Kubernetes pods led to a race condition with AWS's ALB (Application Load Balancer), causing 502 errors and ultimately losing the race against a 300-second ALB deregistration delay. This incident occurred during a rolling deploy of the checkout service, where a 502 error rate spiked from 0.02% to 4.8% for about four minutes.
The root cause was that the checkout app shut down faster than Kubernetes could notify the ALB that the pod was going away. The lack of a SIGTERM handler and a shorter termination grace period for the pod allowed the ALB to continue sending traffic to the pod for up to five seconds after Kubernetes had already marked it as unhealthy.
To prevent this issue, the checkout service needed to implement a graceful shutdown handler to stop accepting new connections before shutting down, and the pod's termination grace period needed to be increased to allow the ALB controller enough time to notice the pod's termination and deregister its IP address.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
