Session Overview
Discover how New Relic tackles real-world data skew in streaming lakehouse pipelines using Apache Paimon.
In this talk, Irene Rodríguez from New Relic’s Streaming Transformation Service team explores the architectural tradeoffs of handling high-throughput CDC data from sources like MySQL, PostgreSQL, and YugabyteDB. Learn why standard static bucketing breaks down under severe data skew, and compare the pros and cons of postponed bucketing, dynamic bucketing, salting, and composite business partitioning to keep compaction healthy and downstream Kafka topics balanced.
What You'll Learn
- The core pipeline: Ingesting database CDC into Apache Paimon before fanning out to Kafka
- Data skew vs. hot partitions: Distinguishing volume imbalances from query traffic intensity
- Logical skew vs. physical skew: Why you must adapt storage infrastructure to real business data
- The limits of static bucketing: Sizing limits, memory pressure, and compaction bottlenecks
- Postponed bucketing in streaming: How delayed staging works and why it impacts end-to-end latency
- Dynamic bucketing & the "eternal bucket" trap: Why high-churn deletion workloads cause bucket proliferation
- Practical mitigations: Manual salting, composite business keys, and downstream balance tips
Chapters
- 00:00 — Introduction & New Relic's CDC ingestion pipeline
- 01:29 — Why Apache Paimon: LSM-tree storage & filtering
- 02:49 — Logical partitions vs. physical buckets
- 06:01 — Data skew vs. hot partitions: Volume vs. traffic
- 07:38 — Logical skew vs. physical infrastructure skew
- 09:46 — Pitfalls of static bucketing under data skew
- 13:20 — Postponed bucketing: Staging directories & latency tradeoffs
- 16:12 — Dynamic bucketing & the eternal bucket problem
- 19:22 — Workarounds: Salting & composite business partitioning
- 21:17 — Key takeaways & summary
