Session Overview
Discover how Pinterest solved high CPU usage and complex Kafka upgrades during cross-cluster replication with Neo Shallow Mirror.
In this talk, Vahid Hashemian from Pinterest shares how Pinterest moved past the "compression tax" of standard Kafka MirrorMaker—which decompresses and recompresses every record—by replicating raw compressed batches directly. Learn how their team used Byte Buddy runtime bytecode interception to bypass patching Kafka core source files, keeping cross-cluster mirroring lightweight, reliable, and easy to upgrade.
What You'll Learn
- The compression tax: Why standard record-by-record mirroring causes CPU spikes and OOM crashes
- Batch-level replication: Replicating entire compressed batches rather than individual records
- Runtime bytecode patching: Using Byte Buddy to modify Kafka client behavior without touching core JARs
- Trade-offs: When to use Neo Shallow Mirror versus MirrorMaker 2
Chapters
- 00:00 — Introduction & mirroring challenges at Pinterest
- 01:34 — The compression tax in standard MirrorMaker
- 03:52 — Identity Mirror: Replicating raw compressed batches
- 05:21 — The upgrade tax & maintaining core source patches
- 07:04 — Introducing Neo Shallow Mirror
- 10:01 — Protocol adjustments & Byte Buddy runtime patching
- 18:48 — Neo Shallow Mirror vs. MirrorMaker 2
- 20:09 — Production impact & key takeaways
