Scale isn’t one number: three ways a Kubernetes control plane runs out of room
Three clusters in trouble, and they look nothing alike. The first is a training fleet. It sat at a few The post Scale isn’t one number: three ways a Kubernetes control plane runs out of room appeared first on The New Stack .
Three clusters faced control plane issues, all different. The first was a training fleet, reaching thousands of nodes before suddenly skyrocketing to fifteen thousand. Kube-apiserver consumed memory, etcd complained. The second cluster never exceeded a few thousand nodes and suffered more drastically. It was an agent platform, with objects constantly created, suspended, resumed, destroyed, resulting in thousands of writes per minute.
The third cluster was neither large nor fast, hosting two hundred operators and tens of thousands of objects, many of which remained idle. AI workloads were two of them, the third arbitrary. They were running out of space in three distinct ways, with fixing one issue not alleviating the others. KubeCon + CloudNativeCon was happening in Salt Lake City from November 9 to 12.
Guidance was scarce, offering advice up to 5,000 nodes, then nothing. Adding more apiservers worsened the problem, as each new apiserver replicated the entire watch cache, consuming memory they didn't have. No magic fork existed; they got there by carefully considering what each store held. The real challenge wasn't scaling, but assembling the right configuration.
Three types of scale to consider: node-bound, churn-bound, and state-bound. Node-bound clusters were large, homogeneous, and faced the constant heartbeat of kubelets. Reaching 100,000 nodes with a 10-second heartbeat generated about 10,000 writes per second, with lease renewals comprising 70% of the writes. State-bound clusters, with many objects mostly idle, had the cost in resident bytes multiplied by the number of apiservers holding them, making the add-an-apiserver reflex counterproductive.
Platform clusters, with various kinds of objects and most of them dormant, had the cost in resident bytes multiplied by every apiserver holding them, making the reflex of adding more apiservers unhelpful. Three seams in the system allowed for potential fixes: the storage contract, node liveness contract, and shard axis. Addressing these seams could help alleviate the control plane issues while maintaining the system's efficiency.
Written by urgent.news from The New Stack's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
