Get the latest tech news
Streaming joins are hard
Why are streaming joins so tricky? Discover the hidden complexities of real-time data, and learn how to tame the infinite stream!
RowlingFantasyUnited Kingdom1Harry Potter1997-06-262George OrwellDystopianUnited Kingdom219841949-06-083Agatha ChristieMysteryUnited Kingdom3Murder on the Orient Express1934-01-01Join resultsWhile this is conceptually very simple, the query planner actually has a number of different execution strategies that it'll use depending on scenario. You "materialize" your data into the derivation SQLite database, and then you can run whatever join queries you want to, albeit with huge caveats about handling deletions and reductions. In summary, streaming joins are difficult because they deal with unbounded data, require efficient state management, and must operate under strict memory and latency constraints.
Or read this on Hacker News