Distributed Locks Are More Dangerous Than They Look

Distributed locks are not simply more powerful versions of local mutexes. They introduce new complexities and hazards that can lead to serious problems. While a local mutex is confined to a single failure domain, a distributed lock spans multiple processes, networks, and storage systems. This means that the lock service may incorrectly determine ownership changes even when the previous owner is still running or paused.

The key question is whether an expired client can still cause unintended effects through its expired authority.

One example is a lease-backed lock used for a reconciliation job. The process acquires a lease, performs the critical operation, and then releases the lease. However, if the process pauses after acquiring the lease, a competing worker may acquire it and execute the job before the original process resumes. There is no guarantee that the resumed process recognizes that its authority has changed. Similarly, delayed network requests or process terminations can also cause similar issues.

Even a perfect lock service does not automatically make resource access safe. Coordination systems can provide strong ownership semantics without controlling the actual resource. For instance, ZooKeeper's lock recipe uses ephemeral sequential znodes, while etcd ties lock ownership to leases that expire. These mechanisms establish ownership inside the coordinator but do not prevent an old process from writing to an external resource after its authority has ended.

Leases trade off between deadlock and stale ownership. While they prevent permanent locks from blocking progress forever, they introduce the risk of time-bounded authority. Redis Redlock, for example, computes a remaining validity window based on elapsed acquisition time and assumes local clocks progress at roughly the same rate. However, this assumes that clocks are accurate and synchronized, which may not always be the case.

Consul also uses TTLs as a lower bound for invalidation, but it warns that expiration may occur later due to clock skew. Failure detection can produce false positives, leading to premature lock release while the owner is still alive. Extending lease durations does not solve the underlying timing issues or make network delays disappear.

The solution lies in fencing tokens, which convert timing problems into ordering problems. A fencing token is a monotonically increasing value issued whenever ownership changes, ensuring that only the latest owner can modify the protected resource. This approach eliminates the timing issues associated with leases and provides a more reliable mechanism for maintaining data consistency in distributed systems.