Zero-downtime credential rotation in OpenTofu: the two-phase pattern (and the two bugs it taught me)
Rotating a long-lived credential sounds trivial: create a new one, hand it out, delete the old one. In practice "hand it out" is never instant. A pipeline that started five minutes ago still has the old token in its environment. A service read its config at startup and caches it. A deployment that pushes the new secret to three systems finishes the first one long before the third. If the old…
Rotating long-lived credentials may seem simple: generate a new one, distribute it, and then remove the old one. However, the real-world implementation is not always instantaneous. For instance, a pipeline that started five minutes ago might still be using the older token. A service might cache its configuration at startup, and a deployment that pushes the new secret to three systems may finish the first one before the third.
If the old credential is revoked as soon as the new one exists, everything in that gap will break. To avoid this, an overlap is needed, where both the old and new credentials are valid while the old one gradually phases out.
This article explains how to implement this overlap using OpenTofu, without relying on a scheduler, Lambda, or external state, and highlights two bugs encountered during the process.
The key concept involves two slots that alternate: Phase 1 and Phase 2. Each slot contains its own independent credential. Consumers receive Phase 2 during the overlap window, which lasts for a fraction (W) of the credential's lifespan (E). A complete cycle lasts for 2E minutes. The variables E (expires_in) and W (rotation_window) determine the duration of each phase and the overlap period, respectively.
Three small resources manage the timeline of a single credential:
1. locals: Holds the expiry and rotation window in minutes, and allows for changing the expiry duration, which restarts the cycle.
2. cycle_timer: Fires once per full cycle (2E) and resets the deterministic_time resource.
3. deterministic_time: Holds a stable timestamp (t0) that remains constant throughout the cycle, regardless of changes in the plantimestamp() function. This timestamp is used to calculate the active phase of the credential.
Rewritten by urgent.news; may contain errors — check the original.
Source: dev.to