Daily Dose of DevOps — Kubernetes readiness vs liveness probes

Kubernetes probes serve as distributed-systems failure detectors with distinct consequences. Readiness probes halt a Pod from Service endpoints when they fail, while liveness probes prompt kubelet to restart the container upon failure. The disparity is crucial because detection is inherently imperfect: tighter thresholds reduce detection time but heighten false positives during transient load.

Readiness probes safeguard traffic, whereas liveness probes trigger process restarts to resolve deadlock. Readiness should answer: "Can this replica safely accept new work now?" It may encompass essential local state and critical downstream dependencies. However, indiscriminate probing of every dependency can trigger a cascade: a single database slowdown could mark all replicas as unready, eliminating capacity when graceful degradation is required.

Liveness probes should answer a more specific question: "Is the process irrecoverably stuck, requiring restart as the best remediation?" It should not fail due to a remote dependency's unavailability. Restarting functional processes during a network partition adds cold-start pressure without rectifying the dependency. Probe configurations: startupProbe checks httpGet on path /health/startup at port 8080 with periodSeconds 5 and failureThreshold 30; readinessProbe checks httpGet on path /health/ready at port 8080 with periodSeconds 5 and failureThreshold 2; livenessProbe checks httpGet on path /health/live at port 8080 with periodSeconds 10 and failureThreshold 3.

The startup probe generates a temporal firewall, suspending liveness checks until initialization completes, thereby avert slow but valid startup from entering a restart loop. Probe timing should be based on actual distributions, not tradition. Set check interval as p, timeout as t, and failure threshold as f. Estimate worst-case detection latency as p * f, plus request timeout effects.

Balance this against service recovery-time objective and the cost of a false positive. Validate under CPU throttling, garbage-collection pauses, dependency latency, and node pressure. Key points: readiness alters routing; liveness initiates restart; startup shields initialization. Maintain local and conservative liveness checks.

Design readiness for graceful degradation instead of dependency-fueled outages. Calculate timings from latency distributions and test them in real resource constraints.