Building a Canary Deployment Pipeline from Scratch: A Practical Guide

A canary deployment is a technique used to gradually release changes in software versions, monitor their impact, and minimize the risk of production incidents. This approach involves running the new version alongside the existing stable version, initially routing only a small percentage of traffic to the new version. By observing metrics and logs, you can determine whether the new version performs as expected before gradually increasing traffic or rolling back to the old version if issues arise.

The term "canary" originates from the historical use of canaries in coal mines as early warning systems for dangerous gases. Similarly, a canary deployment serves as an early indicator of potential issues in a production environment. A typical canary rollout begins with version v1 handling most production traffic, while version v2, the canary, is deployed and receives a small percentage of traffic.

Metrics and logs are closely monitored for a predefined period. If the canary version exhibits no errors or performs well under real traffic, traffic is progressively increased. Conversely, if the canary shows signs of being unhealthy, such as elevated errors or latency, traffic is quickly shifted back to the stable version, and the canary deployment is rolled back.

This method differs from a rolling deployment, which involves Kubernetes gradually replacing old pods with new ones without explicitly analyzing traffic percentages or promoting the new version. Canary deployments offer several benefits, including reduced blast radius, production validation, controlled rollouts, and faster recovery in case of regression.

However, it's important to note that canary deployments do not eliminate deployment risk; rather, they make it easier to detect and contain problems before a full rollout. To implement a canary deployment, Kubernetes Deployments and Services are created for both the stable and canary versions. The stable Deployment runs the current release, while the canary Deployment runs the candidate release.

Separate Services are created for each version to enable independent routing of traffic. The stable Service, for example, is configured with the name "app-stable," while the canary Service is named "app-canary." By following this high-level architecture and utilizing Kubernetes and ingress controllers, software teams can effectively implement canary deployments to enhance the reliability and safety of their production releases.