TL;DR

Stream processors like Kafka Streams need to evolve to address the increasing complexity and scale of modern streaming workloads. Apurva Mehta suggests decoupling compute from storage and leveraging technologies like Kubernetes and object storage to enhance the operational experience. This evolution will enable developers to build more efficient, scalable streaming applications.

Opening

Over the past decade, the adoption of Kafka has skyrocketed, becoming a staple in the infrastructure of companies worldwide. Yet, as these organizations increasingly rely on stream processing for complex, large-scale applications, the tools they use have not kept pace with operational challenges. For instance, while Kafka itself has evolved, stream processors like Kafka Streams remain largely unchanged, struggling to meet the demands of today's cloud-native environments.

What You'll Learn (Key Takeaways)

• Stream processor modernization – Decoupling storage from compute and adopting Kubernetes-native designs can significantly improve the operational management of stream processors.
• Use case alignment – It's crucial to match the right stream processing tool to the appropriate use case, avoiding the common pitfall of forcing one tool to fit all scenarios.
• Modular design approach – Embracing a LEGO-like architecture for stream processors allows developers to assemble the necessary components tailored to specific application needs, enhancing flexibility and efficiency.
• Future of stream processors – There is potential for new, more advanced stream processors that can support next-generation applications, especially as AI and real-time decision-making become more prevalent.

Q&A Highlights

Q: What is the operational view for managing stream processors by a platform team versus a user team?
A: When stream processors are critical to a product, they are typically managed by the application team to ensure clear ownership and responsibility. For less critical applications, a platform team may manage the infrastructure, particularly for nearline use cases.

Q: How do you see stream processors evolving with AI agents?
A: While agents require real-time data processing similar to stream processors, it is unlikely that existing solutions like Flink or Kafka Streams will meet the scale required. A new type of multi-tenant, scalable stream processor is likely needed.

Q: What is the current state of observability for Kafka stream processors?
A: Observability remains a challenge, with many companies deploying their own Grafana and observability stacks. A comprehensive data flow view is often missing, which complicates troubleshooting in large, interconnected systems.

Q: How long until stream processors can support AI-scale applications?
A: Large AI companies may already have proprietary frameworks, but it is uncertain how and when these will become available to the broader market. Existing stream processors are likely insufficient for the massive scale required.