The last-write-wins clock was wrong, and I wrote it

A software developer was working on a note-taking app when an idea struck him during his morning routine. He opened a note on his phone about a sermon, saved it, and then went to his laptop to edit it. Once his phone reconnected, a conflict occurred, lasting for eleven seconds and consuming about four hours of his workweek. This led him to recall a decision he had made eighteen months earlier, stating that the last write wins and the clock decides what is last. This sentence proved to be incorrect in certain situations.

The notes table in the app was simple, using SQLite on each device, with a sync endpoint on the server. The update timestamp was set to the current time when a note was written. Initially, this worked fine as the app was primarily used on a single device. However, as more devices were used, the problem became apparent when two devices synced the same note at the same time.

The "last-write-wins" approach seemed valid at first, as it avoided merging per-field data and kept the feature simple. However, when two devices performed edits within a short window, the older edit written offline would be replaced by the newer one written online, as the server kept the edit with the larger timestamp. Despite having a flaky connection, the user's clock was accurate, and the server's clock was also fine. The issue lay in the ordering between them.

The log showed the conflict, where the phone's edit, made offline, was older in wall-clock time than the laptop's edit but was newer in human time. The phone lost the conflict, replacing the user's carefully written note with an older version. This highlighted the problem with timestamps in offline-first systems, as the timestamp should represent the time an edit occurred, not when it reached the device.

To resolve this issue, the developer implemented a hybrid logical clock (HLC) for each note. The HLC consists of two integer fields, hlc_ts and hlc_count, which are monotonically increasing and updated on every local write and reconciled on every sync. The rules for the HLC are as follows:

1. On a local write, take the current (ts, count) and increment the count.

2. On a received remote write, set ts = max(local_ts, remote_ts), and if they are equal, set count = max(local_count, remote_count) + 1.

3. Compare two HLC values lexicographically, with ties broken by device ID.

This approach still prioritizes the later edit as the winner but defines "later" in a causal sense, based on the actual time of the edit, not just the wall-clock time. The developer migrated all existing rows to seed the HLC values from the previous update timestamp and adjusted the sync endpoint to compare HLC values instead of timestamps. The pull path also needed modification to reject remote rows that had already been seen with a higher HLC value, which caused the developer's initial failure.

Although the HLC effectively resolves the ordering issue, it does not address the problem of discarding the loser's body when a note is edited offline. When a user edits the same note on two devices without internet and reconnects, the second edit is lost entirely. While the release notes state that the most recent version is kept, this admission highlights the inherent limitation of the current system.