Session Overview
Discover why and how Goldsky rebuilt their streaming pipeline engine in Rust to replace Apache Flink for real-time blockchain indexing.
In this talk, Jeff (Co-founder and CTO at Goldsky) shares the architectural journey behind Streamlink, Goldsky's open-source streaming data engine. Learn about the limitations they faced running thousands of Flink pipelines—including cold start latency, heavy JVM overhead, and inter-pod shuffling costs—and how they achieved massive resource savings and zero-copy columnar performance using Rust, Apache DataFusion, and Apache Arrow.
What You'll Learn
- The challenges of real-time indexing across 150+ blockchains
- Why multi-tenant Flink pipelines struggle with cold starts and JVM overhead
- Architecture of Streamlink: In-memory columnar processing with DataFusion and Arrow
- Simplifying state management: Replacing complex snapshotting with relational checkpoint cursors
- Extensibility options: Writing Rust plugins and embedded JavaScript transforms
- Benchmark results: Slashing memory usage, lowering CPU costs, and optimizing ClickHouse batch writes
- Roadmap: Adding sidecar database state without losing instant cold-start times
Chapters
- 00:00 — Introduction & blockchain data indexing challenges
- 01:48 — Pipeline architecture: Turbo backfills & streaming edge
- 04:45 — Why Flink struggled: Cold starts, JVM memory & pod costs
- 07:56 — Designing Streamlink: Moving to Rust & DataFusion
- 10:54 — Pipeline configuration, plugins & embedded JavaScript
- 12:01 — Deliberate tradeoffs: Single-pod scaling & simplified state
- 17:40 — Production benchmarks & cost savings in ClickHouse
- 20:05 — When to choose Streamlink vs. Flink & future roadmap
