The hidden failure paths that affect Kubernetes control plane availability

Kubernetes control planes aim to handle individual failures, but their overall availability relies on the behavior of components during high-pressure situations. At large scale, a single component failure can contribute to an outage through various mechanisms, even if it does not crash outright. For instance, an informer allows a controller to maintain a local view of cluster state, improving efficiency when controllers restart simultaneously.

However, this mechanism can become an amplifier of local defects in the control plane. This article examines three scenarios where such failures occurred.

In the first case, an extension API server experienced issues after a dependency upgrade, leading to invalid schema publication that blocked clients. In the second scenario, a large fleet of DaemonSet agents restarting simultaneously created a request storm that overwhelmed the API server. The third case involved a deep liveness check on a component in the API request path that mistakenly identified a dependency outage as a process failure, causing repeated restarts while the dependency remained unavailable.

These cases are particularly relevant for teams managing extension APIs, controllers handling large object collections, and components in the API request path that depend on extensive validation checks.