Running Doesn't Mean Ready: A Kubernetes Troubleshooting Lab

Kubernetes Pods can be running without being ready to handle traffic. This distinction is crucial during application troubleshooting. A container process may stay alive while the application temporarily cannot manage requests safely. To demonstrate this behavior, I conducted a hands-on Kubernetes experiment using an NGINX Deployment, a ConfigMap, a readiness probe, and a Service. The objective was to make a running Pod become unready, observe Kubernetes' changes, and restore readiness without restarting the container.

Understanding the difference between a Pod's running and ready status is essential. Running indicates that the Pod has been scheduled and at least one container is active. Ready signifies whether the Pod meets its readiness conditions and is suitable for Service traffic. Readiness probes check if individual containers are prepared to handle requests.

Consequently, a Pod can remain running while showing readiness as false. This distinction is vital in production troubleshooting, as a running container does not automatically imply a ready application.

The experiment's architecture consists of a single NGINX Deployment with one replica, a ConfigMap controlling the readiness state, a fictional Secret injected as an environment variable, a readiness probe verifying a mounted file, and a Service exposing the workload. The experiment was conducted in an isolated Minikube environment using Kubernetes version 1.30.0. The ConfigMap initially contains a single value, "ready," while the readiness probe succeeds when the mounted file contains exactly "ready."

Before introducing a failure, we establish a healthy baseline. After confirming the Kubernetes context, creating the namespace, and applying the laboratory manifests, we wait for the Deployment to roll out. Subsequently, we inspect the Pod and record its identity, followed by inspecting the Service endpoints. The expected healthy baseline includes the Pod being Running, Ready, the Service having one ready endpoint, and the container not restarting.

Next, we simulate a readiness failure by modifying the ConfigMap to include a "blocked" status. The mounted configuration file changes from "ready" to "blocked," causing the readiness probe to fail. After waiting for the Pod to become unready, we investigate the Pod's status and Service readiness. The key observation is that the Pod remains Running, but its readiness condition turns false.

Notably, the container process does not need to crash for Kubernetes to stop treating the endpoint as ready. The Service, having only one replica, loses its ready endpoint when the Pod becomes unready. We verify this behavior using the EndpointSlice API, which shows the endpoint readiness condition reflecting the failure.

Finally, we recover the Pod without restarting it. The ConfigMap is patched back to its original "ready" status. After allowing the mounted file to update, the readiness probe succeeds again. Waiting for recovery confirms that the Pod returns to Ready=True, and the Service endpoint becomes ready once more. Importantly, the Pod's UID and restart count remain unchanged, indicating no container restart was necessary.

This demonstrates that Kubernetes can recover a workload by altering its readiness condition rather than recreating the Pod.