Session Overview
Discover how Twilio engineered real-time status aggregations and tracking over messy, high-throughput event streams.
In this talk, Parth and Prunali (Engineers at Twilio) walk through the three toughest challenges they faced building a large-scale broadcast tracking platform on Kafka Streams. Learn why naive counters break under real-world retries and out-of-order events, how off-heap RocksDB memory caused persistent Kubernetes OOM kills, and how blocking I/O calls on stream threads caused catastrophic consumer lag—plus the concrete architectural patterns used to fix each one.
What You'll Learn
- The broadcast fan-out problem: Tracking multi-channel delivery across SMS, WhatsApp, RCS, and push
- Why simple counters fail: Handling out-of-order deliveries, duplicate events, and multi-attempt retries
- State modeling: Tracking lifecycle state and per-attempt progress instead of raw event streams
- Debugging off-heap memory leaks: RocksDB buffer bloat, jemalloc tuning, and shared memory budgets
- Fixing Kafka consumer lag: Moving network I/O, external lookups, and retries off stream processing threads
- Using in-stream metadata events to eliminate slow database queries
- Hard-won lessons for running stateful stream processing in production
Chapters
- 00:00 — Introduction & the broadcast tracking problem
- 03:00 — Architecture overview: Aggregator vs. coordinator
- 05:14 — Why counting fails: Retries, duplicates & out-of-order events
- 07:44 — The fix: Modeling lifecycle state per attempt
- 10:47 — Production crisis: OOM kills & off-heap RocksDB memory
- 12:42 — Memory tuning: Shared cache budgets, bounded records & jemalloc
- 14:57 — The lag spike: Why the coordinator fell behind
- 16:40 — The waiting thread: Removing blocking I/O from stream loops
- 17:40 — Moving lookups into stream metadata events
- 19:29 — Load test results & performance improvements
- 20:39 — Hard-won lessons & key takeaways
