Session Overview
Discover how Pinterest modernised its database ingestion architecture using CDC, Apache Iceberg, and AI agent automation.
In this talk, Owen (Software Engineer at Pinterest) walks through the transition from high-latency daily batch dumps to a real-time CDC streaming pipeline built on Debezium, Apache Kafka, Apache Flink, Apache Spark, and Apache Iceberg. Plus, learn how Pinterest tackled massive onboarding scale by building a dual-agent workflow—combining a customer-facing intake bot and an autonomous orchestrator powered by Model Context Protocol (MCP)—to speed up pipeline provisioning by 10x while eliminating human configuration errors.
What You'll Learn
- The pain points of legacy daily batch dumps: 24h+ data latency, repetitive compute waste, and high maintenance overhead
- Architecture of Pinterest’s next-gen streaming system: Debezium, Kafka sanitisation, and Iceberg storage on S3
- CDC tables vs. Base tables: Append-only change ledgers vs. offline point-in-time state
- Why Spark batch merges remain essential to prevent small file explosion in Iceberg
- Solving scale challenges: Handling floods of onboarding requests without exhausting engineering bandwidth
- Dual-agent architecture: Customer intake chatbot + Claude Code orchestrator driven by Jira states and MCP tools
- Real-world impact: 90% latency reduction and a 10x acceleration in pipeline deployment speed
Chapters
- 00:00 — Introduction & legacy batch ingestion challenges
- 03:32 — Next-gen streaming architecture: CDC, Kafka & Apache Iceberg
- 04:40 — CDC change ledgers vs. Base tables
- 06:18 — Architecture deep dive: Bootstrap, Spark merges & compaction
- 09:09 — Streaming challenges: Schema evolution & partition finalisation
- 11:06 — The onboarding scaling bottleneck
- 14:12 — Dual-agent architecture: Customer intake & Claude Code orchestrator
- 16:15 — Automating provisioning with Jira states & MCP tooling
- 19:47 — Impact & benchmarks: 10x faster pipeline onboarding
- 21:05 — Wrap-up & audience Q&A
