Session Overview
Discover how Airbnb built a unified, declarative data movement and transformation platform to bridge online services, streaming engines, and offline data warehouses.
In this talk, Justin Cunningham and Sai Govada explain how Airbnb transitioned away from a fragmented, multi-step pipeline plumbing to an intent-driven platform (DMT). Learn how developers declare pipelines in simple YAML configurations, how durable saga orchestration automates provisioning across Apache Flink and Apache Spark, and how automated schema evolution and column-level governance eliminate data drift and manual backfilling at scale.
What You'll Learn
- The offline vs. online divide: Overcoming business logic drift and high engineering overhead
- Shifting from imperative scripts to declarative YAML and agent-friendly pipeline definitions
- Architectural overview: Control plane, schema registry, pluggable compute (Flink & Spark), and storage layers
- The 4 core pillars: Orchestration, reliability (Skipper durable workflow engine), extensibility, and automated schema governance
- Coordinating zero-downtime schema evolution and personal data privacy annotations across the pipeline DAG
- Batch vs. streaming runtimes: Managing real-time Iceberg ingestion and offline-to-online key-value swaps
- Hard-won production takeaways: Composability, centralized schemas, runtime abstraction, and shadow runs
Chapters
- 00:00 — Introduction & the online vs. offline data divide
- 01:26 — The declarative vision: From days of manual setup to 2-minute deploys
- 04:11 — Evolution & scale of Airbnb's data movement platform
- 07:37 — Architecture: The 4 layers & control planes
- 10:36 — Pillar 1: Declarative orchestration & Flink Table API
- 14:16 — Pillar 2: Reliability & durable sagas with Skipper
- 15:49 — Pillar 3: Extensibility & modular connectors
- 17:20 — Pillar 4: Automated schema evolution & privacy governance
- 22:10 — Offline-to-online batch pipelines with Spark
- 23:42 — 6 production lessons learned
- 26:55 — Future roadmap & closing thoughts
