Managing Complex Application State with Reactive Data Flows

Complex applications with large data flows can become challenging to manage. As the app grows, so does the complexity of rules that depend on derived values. External data flow in and out of the system, and keeping everything consistent while the user interacts with the UI presents a significant challenge.

However, a set of four components can simplify the problem. Datastar and Glimmer enable the creation of a reactive UI that reacts to changes in the data. Domino acts as a transactional data flow engine that encodes business logic. Ebb provides a clean method to coordinate data flows into and out of the system.

Ebb serves as a bridge between the internal and external data flows. It receives events from outside the system, transacts them in Domino where derived values are calculated, and then Glimmer drives the UI updates based on the resulting state. User input flows in the opposite direction, going from the UI into Domino, getting transacted, and triggering effects that flow out of the system through Ebb.

Ebb essentially functions as a service bus, providing access to external resources such as databases, APIs, email sending, and PDF generation that the application needs.

Despite being based on the Java ecosystem, a port of Missionary has been implemented in pure Clojure. This implementation, Ebb, improves upon the original by allowing functions to park at any call depth, not just at the lexical level. Missionary's core idea is to provide a library for supervised data flow programming where asynchronous effects can be treated as composable values.

The approach avoids memory leaks and state inconsistencies that are common in reactive software development. By linking the lifespan of resources to the period they are needed, it ensures strict boundaries around asynchronous event streams. This model prevents issues like orphaned websockets or zombie threads consuming system resources.

Missionary also uses a bidirectional flow protocol for backpressure management, allowing producer and consumer processes to negotiate and invalidate stale data before expensive recomputations. This model is particularly suitable for Domino, which manages the application's state. The document model in Domino tracks all fields associated with the application's data, and business logic is expressed as context-free functions attached to paths within the document.

When an event occurs, it is fed into the data flow engine, and rules are triggered that produce a new state of the document. Effects can then be triggered to pass data off to the flow layer managed by Ebb. This process can be visualized as an application being a state machine, where an event triggers a cascade of rules that eventually leads to a new state and triggers effects to update the UI.