The Retry That Restored Access
Retries keep distributed systems moving through timeouts and temporary failures. In an access system, though, an old retry can be more dangerous than a failed request. This is Part 2 of Security Infrastructure in Practice, a series about what happens when security design meets production systems. Access had been removed from a deactivated account. The audit trail showed a successful removal. A…
In the series "Security Infrastructure in Practice," a recurring issue is addressed: how distributed systems maintain access even during timeouts and temporary failures. This article focuses on a specific scenario where an access system experienced a problematic retry after an account was deactivated.
The audit trail revealed that the system removed membership from a deactivated account without any intervention from administrators. However, a few minutes later, the membership was restored—without any human action. The problem stemmed from an older provisioning job that had timed out, waited, and then replayed the grant after the account was deactivated.
The sequence of events unfolded as follows:
1. A worker received an event to add a user to a group.
2. The destination accepted the request, but the response timed out.
3. The user was deactivated in the authoritative identity system.
4. The original worker woke up and attempted to retry the old group addition.
Despite the system's components functioning correctly, the final state remained incorrect, highlighting the dangers of relying solely on retries.
To prevent such issues, several recommendations are provided:
1. Events age even when queues do not. A timestamp helps, but clocks and delayed producers make ordering messy.
2. Instead of relying on generation numbers alone, use a generation assigned by the authoritative identity record for more accurate state comparison.
3. Instead of simply retrying the POST operation, implement state-based recovery. Compare the current state with the desired state and make the smallest safe correction.
4. Use stable, immutable external identifiers for account creation and recovery, rather than relying on email or email-like identifiers that can change or be reassigned.
5. Give deactivation work reserved capacity and preserve an inactive record or tombstone long enough to reject old events. Delayed removal can create the illusion of a new identity.
6. Implement a stop condition for retries to prevent infinite loops and allow the system to make decisions when failures cannot be resolved.
7. Measure convergence by tracking the age of the oldest unresolved high-risk identity rather than relying on average metrics. This helps identify when the system is moving toward the desired state.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
