Our own retention policy deleted the version we needed to roll back to

On Friday at 4:10 AM, a release was deployed but quickly resulted in elevated error rates. The decision was made to roll back to a previous version, but the rollback attempt failed after just eleven seconds due to an unknown error. The image that had been running all week had vanished from the registry.

The registry had a retention policy that kept the ten most recent images per repository, based on a time when merges occurred multiple times daily and ten images equated to two weeks of history. However, with the introduction of nightly dependency rebuilds, preview builds, and builds on every merge to the main branch, ten images now only represented approximately thirty hours of history.

On Wednesday night, a rule that had not been activated in two years automatically deleted the version that had been serving production since Monday. This deletion occurred without anyone realizing the policy's existence. Consequently, instead of rolling back, the team resorted to rolling forward to a patch, obtained a review from someone who had already begun the weekend, and endured fifty minutes of degraded service in lieu of a successful rollback.

The key change lay in the deploy pipeline applying an immutable tag to any image promoted to production. The lifecycle rule excluded this tag, while a separate rule retained any image deployed for a year. Preview and branch images, however, expired after seven days, which accounted for the unexpected expense of storage. The images were being deleted despite costing about eleven dollars a month to retain.

Moreover, a scheduled job checked whether the currently deployed image and the previous one were still present and accessible. If either was absent, the job would immediately fail. This rollback rehearsal proved vital, uncovering two additional services whose prior version had already been removed, and one case where the chart values for the previous release had never been recorded at all.

Ultimately, a rollback plan is contingent on the existence of something. In this scenario, the plan was being quietly undermined by a policy written for a repository that no longer mirrored the current one. Sergey Shinder highlighted the importance of having a rollback plan that aligns with the reality of the system in use.