Session Overview
Discover how Adobe protects its large-scale identity graph from oversized clusters and database merge storms using stateful stream processing with Apache Flink.
In this talk, Jianmei Ye and Xinyuan Liang from Adobe explain how bot traffic, incognito browsing, and misconfigurations create giant identity clusters that cause transaction conflicts in FoundationDB. Learn how Adobe moved beyond slow offline batch detection to an in-stream anomaly detector using Flink side outputs, broadcast state, and hybrid state management—reducing detection latency from 15 minutes to under 30 seconds and cutting database conflicts by 65%.
What You'll Learn
- The oversized graph problem: How bot scraping and mock data trigger database merge storms and hot partition bottlenecks
- Why offline batch jobs fail: Why 15-minute batch detection runs too slow to prevent downstream SLA violations
- In-stream detection architecture: Decoupling detection from the hot path using Flink side outputs and broadcasting block decisions
- Hybrid state strategy: Keeping high-volume transient counters in heap memory while snapshotting durable blocklists to checkpoints
- Flink on Kubernetes at scale: Using the Apache Flink Kubernetes Operator and Argo CD for zero-touch declarative upgrades
Chapters
- 00:00 — Introduction & Adobe identity graph architecture
- 04:51 — The problem: Oversized clusters & merge storms
- 07:47 — Why offline batch detection falls short
- 10:05 — In-stream anomaly detection with Apache Flink
- 12:01 — Decoupling the hot path: Side outputs & broadcast state
- 16:14 — State management: Transient in-memory counters vs. durable state
- 19:30 — Production impact & conflict reductions
- 21:14 — Operating Flink on Kubernetes with GitOps & Argo CD
- 25:57 — Hard-won lessons: Classloaders, DAG upgrades & serialization
- 28:37 — Summary & architectural takeaways
