Data Streaming Summit 2024: The Future of Streaming and AI

In just 20 days, the Data Streaming Summit 2024 will kick off, and we couldn’t be more excited! This event marks a significant evolution in our journey, as we transition from the Pulsar Summit into something bigger—Data Streaming Summit. Why the change? Because the challenges and innovations in the data streaming world go beyond any single technology. Whether it’s Apache Pulsar, Kafka, or other messaging or streaming technologies, the ecosystem is expanding, and our summit is designed to reflect that growth.

What We Heard from You

Earlier this year, we had insightful conversations with customers and community members, where they shared their experiences with data streaming workloads. At StreamNative, while many of our customers use Apache Pulsar, we know that Kafka clusters and other messaging or streaming technologies workloads also play a major role in their ecosystems. Despite using different technologies, the pain points were strikingly similar across the board.

1. Scaling data streaming workloads: One of the biggest challenges is scaling data streaming environments, particularly with Kafka, where repartitioning requires a massive effort. For traditional message queue users, many of them struggle with the lack of scalable solutions.
2. Integration with batch processing platforms: There’s a growing demand to seamlessly integrate data streaming with batch processing tools like S3, Databricks, Snowflake, and BigQuery.
3. Generative AI: Across the board, everyone is looking to harness the power of generative AI to make their systems smarter and more efficient.

These discussions highlighted the need for a broader conversation within the data streaming community, which is why we’ve expanded the summit’s scope this year.

Announcing an Exciting Lineup of Speakers

We are thrilled to bring together industry leaders and experts who are shaping the future of data streaming.

• Hugo Smitter, Principal Platform Architect at FICO, will take the stage to talk about how FICO is transitioning its platform engineering team into API-first services. This shift is critical to powering all of FICO’s applications, ensuring scalability and future-proofing their infrastructure.
• Vahid Hashemian, Staff Software Engineer at Pinterest, who is Apache Kafka Committer and PMC member will talk about innovation he led at Pinterest to created Tiered Storage for Kafka.
• Hao Sun and Si Lao Software Engineer at Uber, and Matteo Merli, CTO at StreamNative will talk about Kafka and Pulsar replication in a back to back session.
• Anup Ghatage and Venkateswara Rao Jujjuri at Salesforce will share their hands-on experience scaling BookKeeper from terabytes to petabytes.

You’ll also hear from Sijie Guo and Matteo Merli, the original creators of Apache Pulsar, who will give a sneak peek into the upcoming Pulsar 4.0 release as well as the StreamNative’s core engine URSA’s evolution. They’ll discuss how the data streaming landscape is evolving and what new features and capabilities are on the horizon.

Breakout Sessions to Watch

We’ve organized the breakout sessions into three major tracks: Technology Deep Dive, Ecosystem and Use Cases, and AI. Here are some sessions you absolutely won’t want to miss:

1. Powering Billion-Scale Vector Search at Milvus with Apache Pulsar by Zilliz: Discover how Apache Pulsar is enabling vector search at a massive scale for AI-driven workloads.
2. Scaling Kafka Replication at Uber’s Monumental Scale by Uber: Learn how Uber handles replication challenges as they scale Kafka to support their global operations.
3. Making Kafka Connectors Dance with Apache Pulsar by StreamNative and Google Cloud: This session will explore how Pulsar is enhancing Kafka connectors for seamless data flow and integration.
4. Pinterest Tiered Storage for Apache Kafka by Pinterest: Dive into how Pinterest has optimized their Kafka storage for efficiency and scale.

Rich Data Streaming Ecosystem sponsors

We’re proud to announce that the Data Streaming Summit 2024 is supported by a diverse range of industry-leading sponsors. These companies are at the forefront of innovation in the data streaming, cloud infrastructure, and AI ecosystems, helping to drive the future of scalable, high-performance streaming technologies.

Our sponsors include:

• Google Cloud: Bringing world-class cloud infrastructure and services to empower seamless data flow and AI-driven analytics.
• Snowflake: A leader in cloud-based data warehousing, enabling secure and scalable data integration with streaming environments.
• PingCAP: Innovators behind TiDB, a distributed SQL database, helping enterprises handle massive streaming and real-time analytics workloads.
• Ververica: The creators of Apache Flink, providing powerful stream processing tools that scale across a wide range of use cases.

These partners are not just sponsors—they are collaborators in advancing the future of data streaming technologies. Their contributions ensure that the summit is a comprehensive, high-value experience for all attendees. Be sure to check out their booths and learn more about their innovations during the summit!

A New Era for Data Streaming

With speakers from across the data streaming ecosystem, the Data Streaming Summit 2024 is the place to be if you want to stay ahead of the curve. This event is not just about Apache Pulsar; it’s about the entire data streaming landscape and how we can address common challenges—scaling, integration, and AI. We’re bringing together a community of developers, engineers, and data enthusiasts to share knowledge, experiences, and visions for the future.

If you’re working with data streaming technologies, this summit will offer invaluable insights into where the industry is heading, the tools that will lead the way, and how generative AI is shaping the future of streaming workloads.

So mark your calendars—October 28-29, 2024—and get ready for a deep dive into the future of data streaming! We can’t wait to see you there.

‍