Session Overview
Learn how E6data scales Apache Iceberg to ingest millions of events per second without hitting metadata bloat or commit conflict bottlenecks.
In this talk, Vivek Garg (Chief Product Officer at E6data) breaks down why high-frequency AI event exhausts break standard Iceberg write patterns. Discover how decoupling the horizontally scaled data write path from a dedicated committer service eliminates the commit conflict tax, slashes manifest file bloat, and delivers sub-30-second query freshness at petabyte scale.
What You'll Learn
- The AI exhaust shift: Why agentic systems make 1% downsampling obsolete for debugging and compliance
- Decoupling writes and commits: Separating horizontally scalable data writing from single-committer metadata orchestration
- Eliminating concurrency taxes: Avoiding retry storms and commit lock contention across distributed workers
- Taming metadata bloat: Collating micro-commits to keep query planning under 5 seconds
Chapters
- 00:00 — Introduction & the shift to AI event exhausts
- 02:02 — Why downsampling fails for AI agents & security
- 06:37 — Why Apache Iceberg breaks under high-throughput writes
- 07:28 — Real-world workload: 1.5M events/sec payments use case
- 10:29 — Scaling the accept path: Write-ahead logs & backpressure
- 13:18 — The commit path bottleneck & concurrency tax
- 16:38 — Solving conflicts: Decoupling with a committer service
- 20:01 — Results: Slashing metadata bloat & sub-30s freshness
- 20:48 — Key takeaways & architectural best practices
