StreamNative’s 2023 Year in Review

At StreamNative, we had a productive 2023, improving the developer and operator experiences for our customers, and making it easier than ever to leverage Pulsar’s capabilities. We also continued to collaborate closely with the Apache Pulsar community and reiterate StreamNative’s commitment to open source ecosystems. Let’s conclude 2023 by recapping some of these accomplishments, and look at where we’re going this year.

Features

Kafka on StreamNative (KSN) (Public Preview)

With Kafka on StreamNative (KSN), now in Public Preview, it’s easier than ever for enterprises using Kafka to leverage Pulsar's enhanced capabilities, including multi-tenancy, geo-replication, tiered storage, and unmatched scalability and elasticity. KSN builds on KoP, but contains even more features. To learn more about KSN, check out our  / YouTube Playlist.

StreamNative Private Cloud (Public Preview)

StreamNative Private Cloud, the StreamNative Operator, streamlines the deployment, scaling, and management of self-managed Pulsar clusters. This allows businesses to effortlessly orchestrate intricate data streams, shifting their focus to gleaning valuable insights from their data. Documentation 

Streaming Lakehouse: Pulsar’s Lakehouse tiered storage (Private Preview)

Pulsar's Streaming Lakehouse can integrate with well-known lakehouse storage solutions like Delta Lake, Apache Hudi, and Apache Iceberg, enabling cost savings by using the tiered storage solution of your choice.

Functions on StreamNative Cloud

Pulsar Functions are now available on StreamNative Cloud. Functions enable you to build lightweight, real-time data pipelines for ETL jobs, event-driven applications, and simple data analytics applications.

Functions on StreamNative allow easy debugging via function logs. Sidecar mode enables use of function logs in production with minimum performance impact 

To learn more about using Pulsar Functions on StreamNative Cloud, check out our playlist of Pulsar functions.

StreamNative also provides the SQL-like abstraction - pfsql to enable users to compose filtering/routing/projecting functions. It largely simplifies the development and deployment process for these tasks.

Revocable Cloud API Keys (Public Preview)

StreamNative API Keys offer a dual advantage: they provide a flexible authentication solution compatible with a wide range of clients and enable key rotation at regular intervals to bolster security and compliance.

Broker Autoscaling (Public Preview)

Broker Autoscaling balances the message processing load across brokers, proactively preventing any single broker from becoming a bottleneck. This guarantees the system maintains high throughput and low latency, even when dealing with hefty workloads.

Enhanced Connector Experience

Enhanced Connectors experience gives you the power to develop Connectors tailored to your unique data needs. With the ability to design connectors from the ground up, we're ensuring you will never be limited to the existing IO Connectors we offer. 

Pulsar 3.0 available on StreamNative Cloud

Pulsar 3.0, the next evolution of Pulsar, is now available on StreamNative Cloud. Pulsar 3.0 allows teams to run even bigger workloads, and is easier for developers to work with on their local machines. Updates include LTS support, an improved load balancer, several performance improvements, and Docker images for M1/M2 Macs.

Cluster Metrics now available on StreamNative console

StreamNative Cloud now has a new endpoint and dashboard for Cluster Metrics to provide visibility into your Pulsar clusters and track performance over time. Collect and monitor cluster health and performance metrics in real-time, analyze trends, and proactively maintain your applications to ensure optimal performance. Additionally, you can also view and export your cluster usage metrics from the StreamNative Console.

Rest API

In 2023, we also launched the StreamNative Rest API, allowing users to produce and consume messages with simple API calls, rather than having to use the Pulsar TCP protocol or client libraries to do so. To learn more about the Rest API, check out the documentation.

Oxia

The StreamNative team continued contributing to the open source community by open sourcing Oxia, a scalable metadata store that enables Pulsar to scale to over one million topics. To learn more about Oxia, check out the announcement blog post. And stay tuned for its arrival on StreamNative Cloud.

Certification Programs and Learning Content 

As part of our initiative to grow and empower our community, we launched our Developer Certification program through StreamNative Academy. We also released many videos on YouTube on learning to use Pulsar and StreamNative - check out the playlists posted on YouTube.

Key events recap

Pulsar Summit Europe 2023

Pulsar Summit Europe 2023: This event witnessed a remarkable milestone as over 400 attendees from 20+ countries joined the virtual stage to explore the cutting-edge advancements in Apache Pulsar and the real-world success stories of Pulsar-powered companies. This record-breaking turnout at the Pulsar Summit not only demonstrates the surging adoption of Pulsar but also highlights the ever-growing enthusiasm and curiosity surrounding this game-changing technology. It featured 5 keynotes on Apache Pulsar and 12 breakout sessions on tech deep dives, use cases, and ecosystem talks. They came from companies like Lego, VMWare, Datastax, RisingWave, Axon, Zafin, and others. Watch the sessions.

Pulsar Summit North America 2023

We were honored to organize the Pulsar Summit North America 2023 in San Francisco served as a nexus for the Pulsar community and industry experts to converge, sharing invaluable insights into the future of Apache Pulsar and data streaming. Hosted in person, the summit featured nearly 200 attendees and showcased over 20 enlightening sessions, each a testament to the vibrancy and innovation within the Pulsar ecosystem. They came from companies like Cisco, Discord, Iterable, Attentive, VMware, Flipkart, Boomi, and others.  Watch the sessions.

Partnering with DataBricks

StreamNative partnered with DataBricks on Pulsar-Spark Connector 3.4.1.0. Detailed changes can be found here: https://github.com/streamnative/pulsar-spark/pull/171. Also, check out the Pulsar Spark Connector talk at Pulsar Summit North America!

Collaborated with the Flink community on fixing bugs and contributing the Table API support.

Contributions to the Pulsar ecosystem

• Debezium / Microsoft SQL Server Source - Pulls messages from the SQL Server and persists the messages to Pulsar topics.
• Google BigQuery Source - Feeds data from Google Cloud BigQuery tables and writes data to Pulsar topics.
• Google BigQuery Sink - Pulls data from Pulsar topics and persists data to Google Cloud.
• Snowflake Sink (available through the CLI) - Loads data from Pulsar topics to Snowflake in real-time.‍
• AWS EventBridge Sink Connector Pull data from a Pulsar cluster and persist it to AWS EventBridge, making it easier to build scalable event-driven applications. To learn more, check out our blog post.

Top content of 2023

We provided 50+ new pieces of educational content to help more people understand and get more value out of Pulsar. Here are the top pieces of content enjoyed by the streaming community: 

1. Comparing Pulsar and Kafka: Which streaming technology is right for you?: Compare architecture and features, and learn the best use cases for choosing Apache Pulsar over Apache Kafka.
2. A Comparison of Messaging Platforms: Apache Pulsar vs. RabbitMQ vs. NATS JetStream The tests assessed each messaging platform’s throughput and latency under varying workloads, node failures, and backlogs.
3. Futureproof Kafka Applications and Embrace Pulsar with StreamNative Cloud The most important benefit of the Kafka protocol on StreamNative Cloud is that it allows organizations to harness the strengths of both systems without disrupting their legacy Kafka applications. With a unified event streaming platform, they can take advantage of the features that Pulsar has to offer.
4. Reducing Total Cost of Ownership (TCO) for Enterprise Data Streaming and Messaging In a context of higher cost scrutiny, saving costs free up resources for other strategic initiatives, enhancing the overall efficiency and competitive edge of the organization. Apache Pulsar offers several key advantages to save costs. 
5. How Pulsar’s architecture delivers better performance than Kafka This blog explores the factors contributing to Pulsar's impressive performance despite the architectural differences. 
6. Secure Your Pulsar Cluster with Revocable API Keys A feature that’s only available on StreamNative Hosted and BYOC Pulsar clusters. StreamNative API keys offer both a flexible authentication solution that can work with any client, and a revokable key that can be rotated on a regular interval for security and compliance, or immediately revoked in the event of a security incident.
7. General Availability for Pulsar Functions on All New Clusters in SN Cloud With Pulsar Functions generally available, it’s now vastly easier to do lightweight stream processing on a StreamNative Pulsar cluster. There’s no separate computing cluster you need to set up, you can easily deploy Pulsar Functions using Terraform or pulsarctl, and you can see logs and exceptions directly within the StreamNative Console.
8. Kafka on StreamNative: Bringing Enterprise-Grade Kafka Support to StreamNative Pulsar Clusters (KSN) Excited to present Kafka on StreamNative (KSN), running on Pulsar 3.1, now in Public Preview, an offering tailored to enterprises using Kafka who want to leverage Pulsar's enhanced capabilities.
9. Introducing Extensible Load Balancer in Pulsar 3.0 Thrilled to introduce our latest addition to the Apache Pulsar version 3.0, Extensible Load Balancer, which improves the existing Pulsar Broker Load Balancer. 
10. Enhance Your Compliance and Data Governance with Apache Pulsar and StreamNative In this article, Apache Pulsar, with its robust and flexible features, provides a comprehensive solution to meet the complex requirements of data compliance, even more in highly regulated industries where data lineage, governance, and compliance are critical.
11. A Practical Guide to Enterprise-Grade Security in Apache Pulsar This blog introduced available security combinations in Pulsar and then give some best practices for implementing authentication and authorization.

Looking ahead to 2024

We’re going to be launching a new podcast by the end of March, 2024! The podcast - Sweet Streams - is inclusive of all stream processing and related technologies! We’ll be interviewing community members as well as founders and experts from various streaming software companies and open-source organizations. If you are interested in being interviewed, you can fill out our form here!  

In 2024, we’re excited to continue growing the StreamNative community and empowering our customers to build the next generation of cloud-native streaming and messaging applications. 

Some of our upcoming product enhancements for StreamNative Cloud include Pulsar RBAC, KSN GA, Private Cloud 2.0 GA, Function autoscaling GA, Bookie Autoscaling, Oxia on cloud GA, LDAP integration, Azure support and many more. 

Subscribe to the StreamNative newsletter to stay updated on the exciting announcements we have planned for 2024.

{{cta-blog}}