I Moved Everything to Kubernetes, Then Moved It Back

In a recent development, a reporter recounted their experience of moving six self-hosted services from Docker Compose onto Kubernetes and then back again, while also introducing a seventh service to the cluster. The reporter emphasized that the migration and reversal were not technically challenging, but rather, the issue stemmed from an underlying assumption about the purpose of Kubernetes.

The migration process involved using K3s on a single 4-core, 24 GB machine, which already hosted several compose stacks behind a reverse proxy. The reporter implemented various tools such as ArgoCD, supply-chain policy, Trivy image scanning, SealedSecrets for credentials, and Kyverno admission policy to ensure a smooth transition.

The migration plan focused on maintaining the same external domains and ports, and the reverse proxy remained the anchor throughout the process. The reporter successfully migrated each service individually, ensuring that the public surface never changed while the backend swapped underneath. Once the migration was complete, the reporter proceeded to reverse the process, relocating everything back to Docker Compose within a single day.

Notably, the reporter highlighted that only a few services, such as lab-environment, headlamp, and a namespace called pr-lanes, remained on the Kubernetes cluster. The reporter concluded by emphasizing that the services they migrated were not their own creations, as Kubernetes only provides value for software they owned and controlled.

This realization led the reporter to question the necessity of running Kubernetes for such software, as the infrastructure did not provide any additional benefits compared to using Docker Compose. The reporter also pointed out that their supply-chain policy, which scanned images and enforced critical CVE patches, was built for software they authored, and therefore, enforcing it cluster-wide for third-party software was not practical.