Session Overview
Can columnar execution and native vectorization supercharge stream processing?
In this talk, we dive into the lessons learned from building Iron Vector—an Apache Flink accelerator built with Rust, Apache Arrow, and Apache DataFusion.
What You'll Learn
- Why columnar data layouts matter for streaming throughput (SIMD, cache locality, and data skipping)
- How query plans map from Flink SQL to DataFusion via Substrait
- How zero-copy off-heap memory transfers work across JNI boundaries
- Challenges when adapting a batch query engine to unbounded streaming workloads
- Emerging open-source alternatives, including StreamFusion and Streamlink
Chapters
- 00:00 — Introduction & the case for columnar streaming
- 00:44 — Benchmarking Iron Vector: 2x throughput
- 01:16 — Why columnar data layouts matter
- 02:32 — Native query engines & Apache DataFusion
- 04:36 — Logical vs. physical execution plans
- 06:33 — Mapping Flink SQL plans with Substrait
- 10:05 — Generated physical plans in DataFusion
- 11:38 — Runtime integration & zero-copy memory transfers
- 15:42 — Handling network shuffles across JVM boundaries
- 16:39 — Adapting batch engines to unbounded streams
- 19:50 — Streaming gotchas: timestamps & progress tracking
- 20:31 — What’s next: Streamlink, StreamFusion & open source
- 21:36 — Q&A wrap-up
