Consistency is not a localized property

Consistency is a property that cannot be confined to a single component, as demonstrated by the experience with Apache Kafka. Achieving exactly-once processing in such a system required a significant effort from distributed systems engineers, including the development of new components and a complete rewrite of existing ones (Gustafson et al.

2016). Despite this effort, the creators of Kafka are clear about the limitations of their system: exactly-once processing is an end-to-end guarantee, and the application must be designed to avoid violating this property (Narkhede and Wang 2017).

The implications of this are twofold. First, someone must have a comprehensive understanding of the entire system to ensure consistency. Since no single component can guarantee this property, it falls to someone outside the components to maintain overall consistency. This person must take ownership of the global property and provide a guarantee that it remains intact (source 9e90b30e-f4a).

Second, the responsibility for maintaining global consistency is not something that can be delegated to machines alone. While components may claim that their part works correctly, this does not ensure that the system as a whole remains consistent. The lesson from this experience is that global, valuable properties should not be handed off to machines without a person who is accountable for the entire system and has the guarantee that it functions correctly (Olshan and Liu 2022).

Even after eight years, Kafka's protocol had to be redesigned to address correctness issues that could potentially violate exactly-once processing (Olshan and Liu 2022).