Argo CD Fixed My Drift, Then Deployed My Bad Release

This project began with the goal of running Kubernetes without maintaining an always-on EKS cluster. The author demonstrated this by building a k3s lab on their home server and deploying, updating, and rolling back an application. However, the rollback exposed the issue that Kubernetes restored an outdated version while the saved YAML still declared a more recent version.

To address this problem, the author designed a safer deployment process using Argo CD, an automated GitOps tool. In this process, the CI workflow tested and published an exact image, then stopped at a Git pull request. Git recorded the reviewed version, and Argo CD made the cluster follow that record.

The author tested the design by following one release from source code to running Pods, then tested two opposite failures: Git containing a bad setting while the cluster followed it correctly, and the cluster being wrong while Git was correct. Both tests showed the value and limit of GitOps, as automation could make the cluster match Git but could not decide whether the human-approved version in Git was a good one.

While the CI worker built the release, it did not deploy it. After validation, Buildx created a Linux AMD64 image with the full source commit baked into /version. The CI workflow then updated the Gitops values.yaml with the image digest and opened a pull request. A human reviewed and merged the digest, which was the release instruction.

Argo CD then made the cluster follow Git by repeatedly comparing the version recorded in Git with the version running in the cluster. A pinned, resource-limited Argo CD Core deployment was installed with no permanent API server or public UI. The Application read both the Helm chart and homelab values from the public repository, enabling automated sync, pruning, and self-healing.

The author tested this by changing the cluster behind Git's back, deliberately changing a Deployment to one replica, and observed Argo CD restoring the cluster to match the Git declaration. Then, the author put a bad setting in Git and observed how Argo CD could not correct this by itself. The bad setting was reverted in Git, and Argo CD re-applied the correct version.