Argo CD Canary Deployment: Gating Rollouts with Real SLIs

Argo CD claimed the deployment was healthy, but this only indicated the manifests were successfully applied. After pushing a checkout service change, CI passed, and Argo CD synced cleanly, but the dashboard turned green. Within ninety seconds, an alerting channel was flooded with 500 errors from real customers interacting with the checkout flow.

Upon investigation, it became clear that Argo CD's health check for a standard Deployment only confirmed the pods were running and ready. It had no insight into whether the application was delivering correct responses. With auto-sync enabled and no canary step or automated rollback, a faulty image quickly transitioned from zero to 100% of the pods in a single synchronization cycle.

By the time human operators noticed the surge in error rates, all replicas were already serving the problematic build. To prevent this, the recommended approach is to replace the Deployment with an Argo Rollouts canary deployment. This deployment gradually rolls out to a small traffic portion first, with the rollout gated by real Prometheus Service Level Indicators (SLIs) such as error rate and p99 latency.

Consequently, Argo CD's responsibilities of maintaining cluster synchronization with Git are preserved, while health judgment shifts to Argo Rollouts, where it is more appropriate. Before proceeding with the Rollout, ensure that all prerequisites are met. These prerequisites include having a recent Argo CD (2.x release) and the Argo Rollouts controller, along with the kubectl argo rollouts plugin installed in the cluster.

Additionally, the Rollout CRD must be registered before applying anything referencing kind: Rollout. A metrics backend reachable from within the cluster is necessary for AnalysisTemplates, which can be achieved using Prometheus, Datadog, or CloudWatch providers. A traffic-splitting layer that supports weighted routing is also required, which could be an Istio VirtualService, NGINX Ingress canary annotations, or a Gateway API HTTPRoute.

The trafficRouting block is provider-specific and cannot be mixed mid-tutorial. A pre-existing Argo CD Application pointing to the application's manifests (Helm or Kustomize), currently deploying a plain Deployment, should also be in place. This Application will later be swapped out for the Rollout. Step 1 in the process involves replacing the Deployment with a Rollout.

The pod spec remains unchanged, with the only structural modification being the change from kind: Deployment to kind: Rollout (from the argoproj.io/v1alpha1 API group), and the strategy section transforming into a canary block with explicit steps. It is advisable to begin conservatively, with a setWeight of 10, followed by a two-minute pause, then setWeight of 50 with another pause, and finally setWeight of 100%.

The number of steps and weights should be adjusted according to the acceptable blast radius, not just gut feeling. For example, a service handling payment traffic should have a longer duration than an internal reporting tool. The Rollout configuration should be saved in rollout.yaml. Step 2 involves adding health gates using an AnalysisTemplate.

A pause step alone only provides temporary time; someone still needs to decide whether to promote the rollout. An AnalysisTemplate automates the go/no-go decision based on real metrics instead of relying on a human to monitor a dashboard. The template should define Prometheus queries for error rate and p99 latency, setting successCondition and failureCondition as PromQL thresholds rather than raw request counts, which do not normalize across traffic volume.

The template should reference the Rollout's step list so that it automatically runs at the pause point. The AnalysisTemplate configuration should be saved in analysistemplate.yaml. The most frequent error teams encounter in this process is gating promotion based on infrastructure metrics like CPU or memory, rather than business-facing SLIs.

While these metrics may appear healthy, they may not correlate with the application's correctness. Instead, gate promotion on actual user-perceived metrics, such as error rate, latency, or business events like completed checkouts. Additionally, it is crucial to set the interval deliberately. If the interval is too short, it may result in fluctuations due to naturally occurring variance in metrics.