No official Storm Helm chart — how we put Apache Storm on EKS anyway

The Argo CD application for Apache Storm had been in red status for two days. However, the real problem was more subtle: init containers on Supervisor pods were stuck, repeatedly printing the same line, waiting for the Nimbus component... There was no Nimbus, no workers, and no user interface. The team was attempting to deploy Apache Storm on Kubernetes, a first for the organization. They had never done something like this before, as Storm typically ran on EC2/ASG clusters outside of Kubernetes.

The organization was migrating streaming workloads to the same non-production EKS cluster they were setting up for other applications using Terraform, Karpenter, and Argo GitOps. Running everything together instead of having separate environments was a strategic decision. However, there was no official Helm chart for Apache Storm, and no comprehensive guide on migrating Storm to Kubernetes.

The community had some options, like G-Research's gresearch/storm chart with a Bitnami ZooKeeper subchart, but none of these could be considered a supported product.

The team decided to create a thin local chart, owning the templates and values, rather than relying on an opaque upstream dependency. This was both exciting and daunting, as it was the first time any team in the organization deployed Storm on Kubernetes with no reference architecture or runbooks. They had to evaluate every choice about storage, networking, image choice, and GitOps layout as experimental.

The rollout plan was broken down into phases. First, they focused on getting ZooKeeper and Nimbus running, proving the control plane and ensuring PVCs (Persistent Volume Claims) actually bound. Then, they moved on to Supervisor pods and the Storm UI as stateless Deployments. Next, they added a LogViewer sidecar to Supervisor pods, keeping it from being a separate Deployment.

Finally, they switched to private ECR images instead of Docker Hub, and used a shared Storm namespace in the non-production cluster, rather than creating a dedicated cluster for each environment.

The architecture looked promising on paper, with Component Kind, Storage, and PVC details for each part of the system. However, when the team encountered problems, they were unexpected and hard to trace. The first issue was storage: the chart requested a StorageClass named gp3, but this class did not exist. The cluster only had the legacy gp2 StorageClass, causing the ZooKeeper and Nimbus PVCs to sit in a Pending state.

Without bound volumes, these StatefulSet pods never scheduled, and Nimbus was unable to listen on port 6627, causing downstream processes to block. This led to the Argo app being marked as Degraded.