Kubernetes Troubleshooting: What to Check Before You Restart a Pod
Production incident. Pod is failing. Alerts are firing. The instinctive response is often: "Restart the pod and see if it comes back." Sometimes that restores service — but it can also destroy useful evidence. Before restarting, check these five things. 1. Describe the pod kubectl describe pod <pod-name> -n <namespace> Look at: Container state Restart count Last termination reason Probe failures…
Kubernetes Troubleshooting: Five Checks Before Restarting a Pod
Encountering a failing pod is a common production incident that can be addressed by restarting it. However, this approach may temporarily restore service but also risks losing valuable evidence. To avoid this, follow these five steps before deciding to restart the pod:
1. Examine the pod's details using the command `kubectl describe pod pod-name -n namespace`. Analyze the container state, restart count, last termination reason, probe failures, and events. These details help identify if the issue lies in the application, resources, scheduling, storage, or networking.
2. Review the previous container logs using `kubectl logs pod-name -n namespace --previous`. This is particularly helpful for scenarios where the container has restarted, such as with `CrashLoopBackOff`. The current container might be running again while the actual failure evidence is still present in the previous logs.
3. Investigate why the container terminated. For instance, if it was terminated due to `OOMKilled`, it doesn't necessarily mean you should increase the memory limit. Instead, investigate whether the workload has a memory leak, unrealistic limits, an unexpected traffic spike, or node pressure.
4. Inspect the probes. Even a running container can be unavailable due to failed readiness, liveness, or startup probes. A misconfigured probe can cause an incident even when the application itself is healthy.
5. Prioritize understanding the failure before attempting any fixes. Follow the evidence first: Observe, Capture, Isolate, and then Recover. Capturing the evidence first ensures that you don't lose crucial information before applying a fix. Restarting a pod may temporarily restore service, but understanding the root cause is essential to prevent the same incident from recurring.
This troubleshooting workflow was compiled into the OPSFORGED Kubernetes Incident Troubleshooting Playbook V1.3, a 37-page visual field guide covering various scenarios like CrashLoopBackOff, OOMKilled, probes, DNS, networking, and rollout issues. For a small fee of ₹499 (approximately US$6), you can access this guide to improve your Kubernetes troubleshooting skills and reduce wasted troubleshooting time in production environments.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
