What I Learned Studying EKS Cluster Upgrades (Beyond Just "Click Upgrade")

In the world of cloud computing, Amazon EKS (Elastic Kubernetes Service) has gained popularity due to its ability to simplify the process of running Kubernetes on AWS. However, many newcomers to SRE/DevOps often underestimate the complexity involved in upgrading EKS clusters. After conducting thorough research and delving into real-world postmortems and discussions, the author of this account has learned several crucial lessons that can help others navigate this often challenging aspect of Kubernetes management.

The first key takeaway is that there is no undo button when it comes to EKS cluster upgrades. Unlike many other software upgrades, an erroneous upgrade of the control plane version cannot be simply rolled back. This realization emphasizes the importance of thorough testing before proceeding with the upgrade. The author stresses that testing must be conducted beforehand to ensure that breaking the upgrade is not an option, as the only available fix is to proceed forward rather than backward.

The second critical insight is that Kubernetes frequently deprecates old API versions, with the risk usually not originating from the user's own code. Instead, the issue often lies within Helm charts and third-party tools that have been around for a while and may still rely on outdated API versions. To mitigate this risk, the author suggests utilizing tools like Pluto (pluto detect-helm and pluto detect-files) and Kubent (kube-no-trouble) for detecting potential compatibility issues before they arise.

A lesser-known aspect of EKS upgrades is that addons, such as CoreDNS, kube-proxy, VPC CNI plugin, and EBS/EFS CSI drivers, each have their own version compatibility rules. Upgrading the control plane does not automatically ensure compatibility with these addons. The author advises confirming addon compatibility by checking the version directly using AWS CLI commands, such as aws eks describe-addon-versions, and addressing any compatibility concerns prior to the upgrade.

The fourth lesson revolves around the potential conflicts that can arise from webhooks and operators. These tools, which include cert-manager, service meshes, and policy engines, register ValidatingWebhookConfiguration or MutatingWebhookConfiguration, which intercept every API request. If an incompatible version of these tools is used during an upgrade, the consequences may be confusing and difficult to trace.

The author recommends listing and verifying the compatibility of these webhook configurations before proceeding with the upgrade.

Finally, the author highlights that node upgrades also come with their own set of challenges. One such challenge is the use of PodDisruptionBudgets (PDBs), which can lead to node drain failures if configured too strictly. The author advises using a blue/green node approach, where a new node group is created alongside the existing one, workloads are gradually migrated, and the old group is subsequently removed. This approach is preferred over in-place upgrades and minimizes the risk of unexpected disruptions.

The author concludes by sharing his own study checklist for future upgrades, which includes running Pluto or Kubent against the target version, checking addon compatibility, listing and verifying webhook configurations, assessing PDBs, and considering blue/green node upgrades. This structured approach aims to minimize the likelihood of encountering unexpected issues during EKS cluster upgrades.