Dataflow Model authors grade their own work: what aged well, what missed
The Dataflow Model Revisited
Eleven years after the Dataflow Model paper proposed a unified model for unbounded, out-of-order data, its authors reflect on its legacy upon receiving the VLDB Test of Time award. They find the core foundations—event time primacy, the futility of waiting for completeness, and strong consistency—still sound. However, they admit to over-engineering triggers, entangling windowing with operational concerns, and missing that streams and tables are two views of the same object. The real progress came from database concepts: SQL, incremental view maintenance, and materialized views with freshness contracts. They trace how the completeness principle split into watermarks and snapshot-consistent refresh, and ponder streaming's eventual disappearance beyond analytics.
We focused too much on the mechanics of streaming instead of finishing what the database community started but never completed: making the complexity of analytical streaming disappear almost entirely.
- janpeuker
I used to be extremely into Dataflow/Apache Beam and literally had the paper printed on my desk AND the book. I agree they got Event time versus processing time and Never rely on completeness right, and I love they go deeper into why that was just a hard thing to accept. My head still hurts thinking about unbounded stream triggers and I'm happy we got around to a table-centric model. I still think taking some ideas from Spanner, databases as message bus or consistency information per row, basically in-database CQRS would have been nice, though. Great paper.
- scott_s
I worked in the streaming area for a decade, doing research and development (see: https://scholar.google.com/citations?user=Rdf5OIYAAAAJ&hl=en). After moving on from streaming specifically and moving into the general problems in large data warehouses, I also concluded: just default to SQL for all analytics and the database lens is the best way to think about streaming for analytics.
I still do think that stream programming models are extremely interesting and powerful. But I used to think they would eventually become more mainstream as a way to elegantly program for high throughput, low latency massively parallel systems. That has not been the case, and I no longer think that it will be. People get by with the existing programming languages and models, that seems to be fine.