The Dataflow Model Revisited
Abstract Eleven years ago, the Dataflow Model paper argued that unbounded, out-of-order data was the new normal, and that we must stop waiting for data to ever become complete. It proposed a unified model (windowing, triggers, watermarks, and retractions) for freely trading off correctness, latency, and cost across batch and streaming engines. On the occasion of its VLDB Test of Time award, we…
Eleven years ago, the Dataflow Model paper posited that unbounded, out-of-order data had become the norm, and that we must no longer anticipate data ever reaching completion. It introduced a single model (windowing, triggers, watermarks, and retractions) for easily balancing accuracy, speed, and expense across batch and streaming systems. As the paper celebrates its Test of Time award, this account evaluates its enduring relevance.
The paper's fundamental principles largely hold true: acknowledging event time, rejecting the notion of data becoming complete, and emphasizing strong consistency. However, certain aspects of the analytical interface were misjudged. First, windowing and triggering, with their intertwined operational concerns, were overly emphasized in the article.
Second, triggers were an overly complex solution to a problem that should not have been posed to users. Lastly, the focus on a stream-centric perspective overlooked a fundamental truth: streams and tables are merely distinct ways of accessing the same data with varying access methods.
Ultimately, the practical implementations that fulfilled the analytical objectives of the paper stemmed from the database playbook: SQL, incremental view maintenance, and materialized views with explicit freshness contracts. The author regrets the excessive emphasis on streaming mechanics rather than completing what the database community started but failed to achieve: making the intricacies of analytical streaming nearly invisible.
Nevertheless, the article does not only lament; it also explores how the completeness principle has evolved. Watermarks (where streams remain visible) and snapshot-consistent refresh (where they do not) have both become successful approaches. The latter garnered wider adoption due to its lower demand, and the former can be generalized into explicit change constraints.
Additionally, the article addresses the semantic similarity between batch and streaming debates, the differentiation of low-latency demand along the traditional OLTP/OLAP line, and the continued relevance of analytics with gentler freshness requirements. Finally, it ponders the potential future obsolescence of streaming analytics beyond the realm of data analysis.
Written by urgent.news from Lobsters's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
