We are excited to announce StreamNative Private Cloud, a powerful tool that simplifies the deployment, scaling, and management of Pulsar clusters.
With StreamNative Private Cloud, you can simplify operations and maintenance, including:
Simplified deployment: StreamNative Private Cloud automates the deployment of Pulsar clusters, so teams don't have to worry about manually configuring and managing the individual components.
High Availability: StreamNative Private Cloud sets up clusters in a highly available manner by default. It manages replica placement, broker distribution, and failover mechanisms, ensuring that event streams stay reliable even in the face of failures.
Declarative configuration: StreamNative Private Cloud uses declarative APIs, so teams can define the Pulsar cluster configuration in Kubernetes manifest. This makes it easy to manage the Pulsar cluster and to roll back changes if necessary.
Automated operation: StreamNative Private Cloud supports Auto-Scaling so you can adjust resource allocation in response to incoming workloads.
Cost efficiency: StreamNative Private Cloud supports the Lakehouse tiered storage to offload your cold data to a lakehouse system in Parquet format which brings you cost savings on storage and also supports efficient historical data analysis.
One single operator for a seamless deployment
In 2021, we introduced StreamNative Platform, leveraging StreamNative Pulsar Operators. As we rolled out the initial design of StreamNative Platform, we discovered opportunities to enhance the user experience:
We expanded the support of Pulsar Operators beyond core components to provide a more uniform experience, moving away from the dependence on Helm.
We unified Pulsar Operators to streamline the integration of new components like Oxia, Cloud API Keys, and pfSQL.
We introduced a centralized resource to manage vital configurations, such as authentication.
To address these enhancements, we revamped StreamNative Private Cloud to introduce an all-in-one operator. This new design encompasses the existing ZooKeeperCluster, BookKeeperCluster, PulsarBroker, and PulsarProxy resources along with StreamNative's new components. Furthermore, we introduced the PulsarCoordinator global resource, facilitating cluster-wide high-level configurations. Other components now conveniently refer to the PulsarCoordinator configurations through labels.
Simplified Pulsar services management with a fully supported declarative APIs
On StreamNative Private Cloud, every component is managed through declarative APIs which simplifies your management for Pulsar services - you just have to define the desired state for them. In the earlier version of the StreamNative Platform, Helm was necessary to provision and manage services such as the Console, detector, and toolset. Now, these services can be effortlessly managed through CustomResourceDefinitions.
Enabling the Console, detector, and toolset is as easy as this:
Dynamic Resource Allocation: Pulsar Broker AutoScaling dynamically adjusts resources based on the incoming workload. Whether it's handling a sudden spike in traffic or scaling down during periods of low activity, StreamNative Private Cloud ensures optimal resource utilization, leading to cost savings and improved performance.
Efficient Load Balancing: AutoScaling in Pulsar ensures that the message processing load is evenly distributed across brokers. This prevents any single broker from becoming a bottleneck, allowing the system to maintain high throughput and low latency even under heavy loads.
Cost-Effective Scaling: Traditional scaling methods often result in over-provisioning to handle peak loads, leading to unnecessary costs. Pulsar Broker Autoscaling optimizes resource allocation, ensuring that organizations pay only for the resources they need, making it a cost-effective solution for real-time data processing.
Message Rest API
StreamNative Private Cloud supports a RESTful messaging interface to Pulsar clusters, meaning that you can produce and consume messages without using the native Pulsar protocol or clients. The Rest API supports both non-partitioned and partitioned topics as well as basic and Avro base struct schema.
Example use cases include:
Send data to Pulsar from any frontend application built in any language
Integrate Pulsar with existing automation tools
Ingest Pulsar data into corporate dashboards and monitoring systems
Provide instant access to data in motion for data scientist notebooks
Ingest messages into a stream processing framework that may not support Pulsar
Enterprise-grade security and reliability: Organizations with strict security protocols can confidently use the operators to run Pulsar on OpenShift knowing the operators meet Red Hat’s standards of security and reliability.
Easy installation: Available in the Red Hat Ecosystem Catalog, you can easily install Pulsar Operators in the OpenShift GUI at the click of a button.
Automated operator upgrades: You can automate upgrades for the operators through OpenShift without requiring extra effort to execute the upgrade.
StreamNative Private Cloud is in rapid iteration. Our next key roadmap items include:
Integrate with API Keys: API keys offer both a flexible authentication solution that can work with any client, and a revokable key that can be rotated on a regular interval for security and compliance, or immediately revoked in the event of a security incident.
Integrate with pfSQL: pfSQL is a lightweight SQL-like tool that simplifies real-time data processing built on top of Pulsar functions.
Full support for Kafka on StreamNative (KSN): KSN is an enterprise solution supports KStreams, KSQL, KTables with Topic Compaction, Schema Registry for the Java Client, and Kerberos Authentication for Kafka Clients.
With StreamNative Private Cloud, it's easier than ever to manage your Pulsar Cluster, and you'll get more functionality than ever before.
Eric Shen is a Product Manager at StreamNative. He previously worked at Microsoft & Qiniu & PingCAP & Hikvision mainly focusing on Cloud, Storage, and Databases. He is quietly interested in open source software and distributed systems.