How a Single Misconfigured Liveness Probe Brought Down Our Production Cluster (And How We Fixed It)

On a Tuesday at 2:14 AM, an error in Kubernetes health checks triggered a catastrophic failure in a production video streaming platform. The core issue stemmed from misconfigured Liveness and Readiness Probes that checked external dependencies like the database. When traffic surged by 300%, the database connection pool became saturated, causing the Readiness Probe to time out.

Kubernetes then mistakenly restarted pods, leading to a self-destructive loop where containers kept failing their probes and being killed. The solution required decoupling health checks from external dependencies, tuning timeouts and thresholds, and implementing Startup Probes for slow-starting applications. The incident highlighted the importance of proper health check configurations in Kubernetes to prevent unintended cascading failures.