Why Timestamps Lie in Distributed Systems (and How Logical Clocks Fix It)

Many services store replicated data using a simple function to determine which write to keep: def resolve(incoming, stored): # keep whichever write is newer return incoming if incoming.timestamp > stored.timestamp else stored This code works well in local testing environments. However, it can fail when deployed to production. A customer reported that a profile change they saved was reverted, and there were no errors in the logs.

The issue was that the incoming and stored timestamps were generated on two different machines, with one machine's clock being about 90 milliseconds ahead of the other. The later write carried the smaller timestamp, so the older data was kept instead.

This problem is known as last-write-wins (LWW), and it can silently corrupt a database. The same issue appears in different forms: expired cache entries, distributed traces showing responses arriving before requests, rate limiters resetting mid-window, and deduplication windows dropping legitimate events. The root cause is the assumption that timestamps from different machines can be compared. They cannot, at least not with the precision needed for proper conflict resolution.

To address this problem, we can use the happened-before relation, Lamport clocks, vector clocks, and hybrid clocks used in modern databases. Most of these solutions are already present in the systems we use daily.

Clocks drift over time due to various factors, such as the precision of commodity quartz oscillators and Network Time Protocol (NTP) corrections. NTP can step the clock forward or backward during network partitions, VM migrations, and long pauses, leading to negative durations and reordered events. Virtualization, containerization, and laptops further complicate the issue by causing clocks to pause, resume, or change unpredictably.

The dangerous part is that clocks disagree the most during partitions, failovers, and overload, which are the moments when conflict-resolution code runs hardest. Trusting cross-machine timestamp comparisons in a correctness path is risky, as the clock is least trustworthy at the precise moment our logic relies on it most.

Monotonic clocks, such as time.monotonic() in Python, never run backward and are useful for measuring local durations and timeouts. However, they are meaningless across machines and across reboots because their zero point is arbitrary. To order events globally, we need something else: the happened-before relation.

Leslie Lamport's 1978 paper introduced this concept, which states that in a distributed system, the meaningful order of events is causal, not temporal. Event A happened-before event B if they are on the same process and A came first locally, or if A is the sending of a message and B is the receipt of that message. If no such chain exists in either direction, the two events are concurrent. This means that physical time has nothing to do with the order of events; it's the causal chain that matters.

Lamport clocks are a cheap way to implement this causal order. They use a single integer per process, incrementing it for each local event and sending an incremented value with outgoing messages. When receiving a message, the local clock is updated to the maximum of its current value and the sender's stamp, then incremented by one. This ensures that if event A happened-before event B, then L(A) ≤ L(B). Ties can be broken using a process ID.

By using Lamport clocks or similar mechanisms, we can track causal relations and ensure that our invariants hold true, even when clocks disagree. This approach respects causality and helps prevent silent data corruption caused by clock skew.