{
  "id": 7376379,
  "title": "Upgrading a Locked-Down Linux Fleet with Nothing but kubectl",
  "url": "https://urgent.news/2026/09/14/upgrading-a-locked-down-linux-fleet-with-nothing-but-kubectl",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-14T18:39:04.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/sidharth_cp/upgrading-a-locked-down-linux-fleet-with-nothing-but-kubectl-3d7k"
  },
  "original_language": "en",
  "account": "When operating a fleet of Linux machines using an immutable OS, the root filesystem is read-only and upgrades are applied as a single artifact. While this setup offers great reliability, it raises the challenge of performing routine tasks like rotating login passwords or rolling out new OS versions across all nodes. SSH-ing into each machine and doing these tasks manually is impractical on a large fleet due to the lack of scaling and the risk of drift. To address this, a solution was found that allows changing a password or pushing a full OS upgrade to the entire fleet by applying a single Kubernetes ConfigMap, without the need for SSH, per-node scripts, or risking drift.\n\nTo rotate the password across all nodes, the command `kubectl get configmap kairos-config -n kube-system -o yaml` is used to retrieve the ConfigMap, and then `sed` is employed to modify the `passwd: kairos` entry to `passwd: something-better`. The modified ConfigMap is then applied to the cluster using `kubectl apply -f -`.\n\nSimilarly, to roll out a new OS image to the whole fleet, a ConfigMap named `kairos-upgrade` is created with the new image and version details using `kubectl create configmap kairos-upgrade --from-literal=image=docker.io/you/kairos-rhel96:v2 --from-literal=version=v2.0.0 -n kube-system`. The `-dry-run=client` option is used to preview the changes without actually applying them, and finally, `kubectl apply -f -` is used to apply the ConfigMap to the cluster.\n\nThis entire operator experience revolves around Kubernetes ConfigMap applications, providing an efficient way to manage node updates without compromising the immutable OS model. By separating the cluster-side unprivileged DaemonSet pod, which has limited capabilities and access to specific host directories, from the host-side root-owned systemd service, `kairos-agent-watcher`, the security risk is mitigated. The cluster-facing component can only modify ConfigMaps, while the host-side process performs privileged tasks like applying configurations, flipping the A/B slot, running upgrades, and rebooting nodes. This separation ensures that even if the DaemonSet pod is compromised, the attacker cannot gain root access to every node, as the privileged work is isolated on the host side.\n\nAn additional step is taken during the installation process to ensure that the OS partitions are correctly placed on the appropriate disk. On multi-disk systems, the script selects the smallest attached disk over 20GB that isn't the boot media to prevent the OS from installing over valuable data or the installation media. However, this process can be complicated by leftover Kairos partition labels from previous installations, which could lead to the OS spreading itself across multiple disks. To avoid this, any stale Kairos labels are removed from the candidate disks using `wipefs` and `sgdisk --zap-all` before the installation begins. This ensures that the installer will not incorrectly adopt partition labels from previous installations, maintaining the integrity of the new OS deployment.",
  "summary": "Say you're running a fleet of machines on an immutable operating system — root filesystem read-only, nothing drifts, and you upgrade the whole OS as a single artifact instead of patching it in place. Great for reliability. But now you need to do something mundane: rotate a login password across every node, or roll out a new OS version. The obvious move is to SSH into each box and do it by hand.…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}