When Two Systems Disagree About the Truth, Both Get Believed

In a world where multiple systems coexist and share data, a perplexing issue arises when two or more systems report conflicting information. This phenomenon often occurs without anyone noticing, as each system appears fully functional on its own. The CRM might claim a customer is on a premium plan, while the billing system insists they are on the free tier. Support may trust one system, while finance relies on the other, leaving the customer caught in the middle of a disagreement.

This situation stems from the fact that data is frequently copied, and every copy necessitates maintaining synchronization with the original. When a field is duplicated into a second service to expedite lookups, a nightly job syncs records between two databases, or an integration writes the same fact in two locations, a second source of truth is inadvertently created.

Even though each duplication may seem reasonable at the time of implementation, as time passes, these copies inevitably drift apart. There is no definitive method to determine which copy is correct, as both were designed to be authoritative.

The key to resolving this issue lies in explicitly determining where each critical fact resides and treating all other copies as mere duplicates, subordinate to the source and not allowed to be edited independently. When the billing system and the CRM disagree about a customer's plan, there should be no ambiguity regarding which system is the rightful owner of that fact. Ownership should be clearly established, and every other copy should acknowledge this ownership rather than attempting to edit it independently.

Ambiguity surrounding ownership is the root cause of the disagreement, and the solution is not to build yet another reconciliation job to address the divergence but rather to remove this ambiguity altogether. Reconciliation jobs, in fact, indicate that the battle has already been lost, as they merely clean up the divergence between two systems on a scheduled basis, hoping that the window between drift and repair never coincides with a critical customer at an inconvenient time.

Ultimately, the resolution lies in deciding where the truth resides and ensuring that all other systems defer to it. Two confident systems that disagree do not average out to a reasonable result; instead, they provide everyone with the permission to believe in whichever system they initially trusted.