Two Sources of Truth Will Always Disagree

In a system, the same fact is often stored in multiple places. These include a count in a column, rows being counted, an order status, payment status, an email in a profile service, and the same email in a mailing list. While these figures may align at first, they inevitably diverge over time. This discrepancy is not necessarily a bug; it merely requires one write operation that succeeds and another that fails, one unattended job at an off-peak hour, or a migration that only affected part of the data.

As a result, the two numbers become distinctly different, causing confusion for the report, the customer, and system administrators alike. The system must determine which source is accurate, as both are considered the "truth."

One field may hold the accurate figure, while the others either derive the information or request it, caching it with a clear expiration date and name. For example, a "total" field is a claim, whereas a "cached_total_at_midnight" is a confession. Confessions are easier to debug. Sometimes, duplicates cannot be avoided, such as when two services or databases access a boundary beyond one's control. In such cases, it's essential to acknowledge the situation and address it promptly.

A reconciliation job should be written on the same day as the initial copy, not after the first discrepancy is discovered. It's crucial to report the difference before making any corrections, as hiding the issue may prevent the identification of how frequently and from where the drift occurs. The most troubling scenario is a duplication that nobody designed, such as a column being denormalised for performance reasons in 2022.

While it initially improved speed, it quietly became a second truth that nobody agreed to maintain. Every duplicated fact implies a commitment to maintaining two pieces of code in sync throughout the system's lifespan. However, even the most carefully executed systems will eventually experience drift. Therefore, storing data once and acknowledging the inevitable copies is the most practical approach.