.png)
TL;DR
The session addressed the challenge of integrating real-time event streams into the AI-driven era, focusing on cost-effective data streaming and the unification of streams and tables. The solutions presented included leveraging Apache Flink 2.0 and Pulsar to enhance real-time processing capabilities and simplify migration from Kafka. The key benefit achieved is the facilitation of intelligent, autonomous AI systems that operate efficiently at scale.
Opening
Imagine a world where real-time data streaming isn't just a costly addition to your tech stack, but the backbone of intelligent autonomous systems. As we stand at the cusp of a new era of data streaming, cost pressures from legacy systems and the need for seamless integration with AI-driven applications highlight the necessity for innovation. StreamNative's vision of "Agentic AI" captures this shift, aiming to create intelligent agents that learn and act autonomously, powered by a continuous stream of data.
What You'll Learn (Key Takeaways)
- Integrating AI with Real-Time Data – Discover how the latest advancements in Apache Flink 2.0, with features like disaggregated state management, are designed to meet the demands of cloud-native environments and AI applications, making stream processing more efficient and cost-effective.
- Unified Data Streaming Architecture – Learn how Pulsar's innovative approaches, such as the Ursa Engine and Universal Linking, enable seamless migration from Kafka and facilitate the creation of a real-time lakehouse, unifying streams and tables for coherent data management.
- Real-World Applications of Pulsar – Gain insights from Q6 Cyber’s journey in restructuring their data architecture around Pulsar to handle 75 billion+ records efficiently, overcoming challenges in serialization and scaling while maintaining a robust and flexible system.
Q&A Highlights
Q: Would it be possible to use the Pulsar Client in combination with the Ursa Engine?
A: Yes, it is possible to use the Pulsar client with the Ursa Engine. While there are still some features being finalized, such as compaction and transactions, the integration is underway to ensure full compatibility.
Q: Is Ursa a replacement for Pulsar?
A: No, Ursa is not a replacement for Pulsar. It is an augmentation that provides additional functionalities like multi-protocol support and new storage strategies, enhancing Pulsar's capabilities.
Q: How does the LLM context design handle multi-agent memory management at scale?
A: The LLM context acts as a wrapper for Pulsar functions, using distributed state storage for long-term memory with built-in policies for memory management to ensure efficient use at scale.
Q: What future plans does Q6 Cyber have for Pulsar?
A: Q6 Cyber plans to redesign their reporting applications to be more real-time focused, leveraging Pulsar's capabilities to deliver data insights more swiftly and efficiently to their clients.
Sijie’s journey with Apache Pulsar began at Yahoo! where he was part of the team working to develop a global messaging platform for the company. He then went to Twitter, where he led the messaging infrastructure group and co-created DistributedLog and Twitter EventBus. In 2017, he co-founded Streamlio, which was acquired by Splunk, and in 2019 he founded StreamNative. He is one of the original creators of Apache Pulsar and Apache BookKeeper, and remains VP of Apache BookKeeper and PMC Member of Apache Pulsar. Sijie lives in the San Francisco Bay Area of California.
Matteo is the CTO at StreamNative, where he brings rich experience in distributed pub-sub messaging platforms. Matteo was one of the co-creators of Apache Pulsar during his time at Yahoo!. Matteo worked to create a global, distributed messaging system for Yahoo!, which would later become Apache Pulsar. Matteo is the PMC Chair of Apache Pulsar, where he helps to guide the community and ensure the success of the Pulsar project. He is also a PMC member for Apache BookKeeper. Matteo lives in Menlo Park, California.
Kundan is a Staff Product Manager at StreamNative, where he spearheads StreamNative Cloud, Lakehouse Storage and compute platform for connectivity, functions, and stream processing. Kundan also leads Partner Strategy at StreamNative, focusing on building strong, mutually beneficial relationships that enhance the company's offerings and reach.
Neng Lu is currently the Director of Platform at StreamNative, where he leads the engineering team in developing the StreamNative ONE Platform and the next-generation Ursa engine. As an Apache Pulsar Committer, he specializes in advancing Pulsar Functions and Pulsar IO Connectors, contributing to the evolution of real-time data streaming technologies. Prior to joining StreamNative, Neng was a Senior Software Engineer at Twitter, where he focused on the Heron project, a cutting-edge real-time computing framework. He holds a Master's degree in Computer Science from the University of California, Los Angeles (UCLA) and a Bachelor's degree from Zhejiang University.
Dustin Nest specializes in the creation of engaging technical training content and is StreamNative's full-time Technical Trainer. He brings to StreamNative over seven years of experience in software technical support, debugging, and training content development. He has both a BS and PhD in Chemical Engineering, with additional training and experience in C++, C#, Java, JS, and React. Dustin is excited to be your guide as you learn how to build scalable messaging and streaming applications using Apache Pulsar.
Xintong is an Apache Flink PMC member. He is also a Staff Software Engineer and leads the Flink Shuffle & SDK team at Alibaba. Prior to that, he received a Ph.D. degree in computer science from Peking University.
Jeff is responsible for leading and setting the vision for our technology and oversees all aspects of technology operations. Jeff began his career with the United States Secret Service’s Cyber Intelligence Section, where he focused on attribution of sophisticated cybercrime targeting the U.S. financial infrastructure. Jeff subsequently joined the United States Marine Corps as a Signals Intelligence Officer, and led the collection, analysis, and delivery of critical intelligence products through two deployments. Jeff was then selected to serve as a Branch Chief at the National Security Agency (NSA), where he led a number of classified programs developing state-of-the-art cyber intelligence collection technologies. At IBM Security, Jeff led a multinational distributed team delivering 24/7 SIEM monitoring to a diverse array of large complex corporate network environments.
Daniel is a 36 year old Atlanta native who has been programming backend java applications for the past 15 years. He got his career started in Portland, OR, working as a software support engineer. At Q6, Daniel leads a team of developers responsible for all of the data ingest, enrichment, tranformation, and reporting that needs to get done in order to deliver actionable intelligence to Q6's clients.
Newsletter
Our strategies and tactics delivered right to your inbox