Would your Kubernetes cluster survive losing one ESXi host? I built a read-only tool to find out
Kubernetes sees nodes . vSphere sees VMs on hosts . Neither tells you when DRS has put two of your three etcd VMs on the same ESXi host. From then on, one host failure takes your control plane down, and every dashboard still says green. I spent nearly seven years in VMware support, and I couldn't find a tool that joins those two views and simulates host failures against Kubernetes semantics. So I…
Kubernetes clusters can be vulnerable if an ESXi host fails, potentially taking down the control plane. A read-only tool called kube-hostfail was created to simulate host failures against Kubernetes semantics to identify such vulnerabilities. The tool joins Kubernetes node and etcd-member data with vCenter VM and host data, then simulates single and two-host failures.
It checks etcd quorum, API availability, workload capacity, and DRS anti-affinity rules. If any single host failure could take down the cluster, kube-hostfail exits with a code indicating the level of failure. The tool is designed to be installed in a production environment without modifying the existing cluster. It includes various components such as Prometheus, Grafana, Istio, Jaeger, Kiali, and a gateway API, all in new namespaces with sidecars injected only into the demo namespace.
The tool has been verified through unit tests, end-to-end tests, chart and manifest validation, and a full install on a real cluster. However, it has not been verified in a live production environment and does not model datastore/PV placement, network partitions, DaemonSets, or vSphere HA admission control. Users are encouraged to try the tool and report any issues or suggestions for improvement.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
