Helm Rollback Strategy: Safe Values Promotion in Production
Originally published on kuryzhev.cloud One bad helm upgrade and your production values file is history — literally — unless you have a helm rollback strategy that survives concurrent deploys, stateful side effects, and plain human error. I've watched a team roll back to what they *thought* was the last good revision, only to land on a "superseded" release with a broken feature flag still baked…
In order to ensure safe production values in Helm-based deployments, it is crucial to have a robust rollback strategy in place. This strategy should account for concurrent deploys, stateful side effects, and human errors. The key to a successful rollback strategy lies in pinning exact chart versions and snapshotting values before every promotion.
Before every Helm upgrade, the values.yaml file should be retrieved using `helm get values <release> -o yaml` and stored alongside CI artifacts, tagged with the git commit SHA and release revision number. This ensures that the live cluster state is compared with the deployed values, eliminating confusion caused by drift in live cluster state from the chart repo.
Rollback should be performed by referring to the exact revision number, not just the previous revision assumption. The command `helm rollback <release> <revision>` should be explicitly run, and it is essential to confirm the revision status via `helm history <release>` before proceeding. This prevents accidents caused by reverting to superseded releases or failed deploys.
Additionally, it is crucial to wait for a specified time frame after initiating a rollback, allowing Kubernetes pods to stabilize and ensure a successful rollback. Failing to do so might result in a brief success notification followed by a real outage.
Helm's default history limit may be insufficient for frequently updated services. It is recommended to increase the --history-max value for critical services, keeping in mind the storage tradeoff. Revisions beyond the set limit can be purged when decommissioning releases to maintain etcd performance.
Before rolling back, it is advisable to use the `helm-diff` plugin to visualize changes and run a dry-run with debugging information. This helps identify potential issues such as hook ordering or immutable field problems that could affect the cluster post-rollback.
Rollback does not automatically resolve stateful side effects like database migrations or PVC data. Manual intervention may be necessary to ensure the system remains in a consistent state. Furthermore, it is important to promote values through environment-specific files rather than through manual edits to maintain version control and reduce human error.
Lastly, testing the rollback process itself is as important as the deploy process. Regular rollback drills in a staging environment help validate the rollback strategy, ensuring that the service can be restored without the need for manual intervention. Implementing concurrency controls in CI/CD pipelines can prevent race conditions between rollbacks and fresh deployments, ensuring a smooth and secure rollback process.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
