Streaming at the Crossroads: Can One Architecture Rule Them All?
From Kafka to Pulsar to LinkedIn’s new NorthGuard architecture, the data streaming ecosystem has never been more diverse—or more innovative. AI-driven applications, mission-critical transactional workloads, and massive-scale personalization pipelines are all placing unique demands on streaming systems. At the same time, the industry faces big questions: Will the future be defined by a single dominant architecture, or by a mosaic of specialized systems?
In this panel, leaders from Uber, OpenAI, PayPal, Redpanda, and StreamNative will share their perspectives on the trade-offs between unification and specialization. This panel explores whether one architecture can realistically serve all needs—or whether the future remains multi-technology. We’ll examine a few major topics: (1) latency-optimized vs. cost-optimized design, (2) object-storage-first (S3-backed), leaderless/diskless architectures that unify streaming with the lakehouse, and (3) new requirements raised from AI and agents.
