Session Overview
Discover how Zeta Global achieved order-preserving, fair multi-tenant consumption from shared Apache Kafka partitions using SlateDB and object storage.
In this talk, Ananth Packkildurai (Principal Architect at Zeta Global) explains how to eliminate head-of-line blocking across thousands of tenants without complex sharding or per-tenant topics. Learn how embedding SlateDB inside Kafka consumers creates independent tenant pointers, disables write-ahead log overhead by leveraging Kafka itself, and balances low-cost memory caching with durable S3 spills.
What You'll Learn
- Multi-tenant fairness: Solving noisy-neighbor bottlenecks across 2,000+ tenants sharing Kafka partitions
- Order preservation: Maintaining strict causal event ordering per tenant during traffic bursts
- Embedded LSM architecture: Using SlateDB on S3 as an intermediate buffer without extra coordinator services
- Cost & performance tuning: Bypassing SlateDB's WAL in favor of Kafka and serving bursts from memory before spilling to disk
Chapters
- 00:00 — Introduction & the multi-tenant ordering challenge
- 03:24 — Head-of-line blocking in shared Kafka partitions
- 05:18 — Why per-tenant topics and pod sharding fail to scale
- 07:58 — The single-commit watermark limitation in Kafka
- 10:33 — Embedding SlateDB as an intermediate buffer
- 14:18 — Cost optimization: Disabling WAL & tuning S3 flushes
- 17:49 — Consumer architecture & bounded spill handoff
- 20:26 — Fair scheduling with token bucketing
- 25:37 — Shifting metrics: Kafka offset lag vs. backlog age
