Introducing StreamNative Private Cloud

We are excited to announce StreamNative Private Cloud, a powerful tool that simplifies the deployment, scaling, and management of Pulsar clusters.

With StreamNative Private Cloud, you can simplify operations and maintenance, including:

• Simplified deployment: StreamNative Private Cloud automates the deployment of Pulsar clusters, so teams don't have to worry about manually configuring and managing the individual components.
• High Availability: StreamNative Private Cloud sets up clusters in a highly available manner by default. It manages replica placement, broker distribution, and failover mechanisms, ensuring that event streams stay reliable even in the face of failures.
• Declarative configuration:  StreamNative Private Cloud uses declarative APIs, so teams can define the Pulsar cluster configuration in Kubernetes manifest. This makes it easy to manage the Pulsar cluster and to roll back changes if necessary.
• Automated operation: StreamNative Private Cloud supports Auto-Scaling so you can adjust resource allocation in response to incoming workloads.  ‍
• Cost efficiency: StreamNative Private Cloud supports the Lakehouse tiered storage to offload your cold data to a lakehouse system in Parquet format which brings you cost savings on storage and also supports efficient historical data analysis.

One single operator for a seamless deployment

In 2021, we introduced StreamNative Platform, leveraging StreamNative Pulsar Operators. As we rolled out the initial design of StreamNative Platform, we discovered opportunities to enhance the user experience:

• We expanded the support of Pulsar Operators beyond core components to provide a more uniform experience, moving away from the dependence on Helm.
• We unified Pulsar Operators to streamline the integration of new components like Oxia, Cloud API Keys, and pfSQL.
• We introduced a centralized resource to manage vital configurations, such as authentication.

To address these enhancements, we revamped StreamNative Private Cloud to introduce an all-in-one operator. This new design encompasses the existing ZooKeeperCluster, BookKeeperCluster, PulsarBroker, and PulsarProxy resources along with StreamNative's new components. Furthermore, we introduced the PulsarCoordinator global resource, facilitating cluster-wide high-level configurations. Other components now conveniently refer to the PulsarCoordinator configurations through labels.

Simplified Pulsar services management with a fully supported declarative APIs

On StreamNative Private Cloud, every component is managed through declarative APIs which simplifies your management for Pulsar services - you just have to define the desired state for them. In the earlier version of the StreamNative Platform, Helm was necessary to provision and manage services such as the Console, detector, and toolset. Now, these services can be effortlessly managed through CustomResourceDefinitions.

Enabling the Console, detector, and toolset is as easy as this:

apiVersion: k8s.streamnative.io/v1alpha1
kind: Console
metadata:
  name: sn-private-console
  namespace: pulsar
  labels:
    k8s.streamnative.io/coordinator-name: private-cloud
spec:
  image: streamnative/private-cloud-console:v2.3.3
  webServiceUrl: http://brokers-broker:8080

apiVersion: k8s.streamnative.io/v1alpha1
kind: PulsarCoordinator
metadata:
  name: private-cloud
  namespace: pulsar
spec:
  image: streamnative/private-cloud:3.0.1.4
  detector:
    serviceEndpoint:
      pulsarServiceURL: pulsar://brokers-broker:6650
      webServiceURL: http://brokers-broker:8080
  toolSet:
    enabled: true
    replicas: 2

You can also install the function mesh operator and pulsar-resources-operator to manage functions, connectors and Pulsar resources through declarative APIs which bring you the fully cloud native usage experience with Pulsar.

AutoScaling

StreamNative Private Cloud supports AutoScaling to automatically adjust the available resources of your deployments, and eliminates the need for scripts or manual updates to make scaling decisions.

You can start the AutoScaling by specifying a range of minimum and maximum nodes that your Pulsar cluster can automatically scale to:

apiVersion: pulsar.streamnative.io/v1alpha1
kind: PulsarBroker
metadata:
  name: brokers
  namespace: pulsar
  labels:
    k8s.streamnative.io/coordinator-name: private-cloud
spec:
  image: streamnative/private-cloud:3.0.1.4
  replicas: 4
  zkServers: zookeepers-zk:2181
  pod:
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
    securityContext:
      runAsNonRoot: true
  autoScalingPolicy:
    minReplicas: 1
    maxReplicas: 4

This feature brings the following benefits:

• Dynamic Resource Allocation: Pulsar Broker AutoScaling dynamically adjusts resources based on the incoming workload. Whether it's handling a sudden spike in traffic or scaling down during periods of low activity, StreamNative Private Cloud ensures optimal resource utilization, leading to cost savings and improved performance.
• Efficient Load Balancing: AutoScaling in Pulsar ensures that the message processing load is evenly distributed across brokers. This prevents any single broker from becoming a bottleneck, allowing the system to maintain high throughput and low latency even under heavy loads.
• Cost-Effective Scaling: Traditional scaling methods often result in over-provisioning to handle peak loads, leading to unnecessary costs. Pulsar Broker Autoscaling optimizes resource allocation, ensuring that organizations pay only for the resources they need, making it a cost-effective solution for real-time data processing.

Message Rest API

StreamNative Private Cloud supports a RESTful messaging interface to Pulsar clusters, meaning that you can produce and consume messages without using the native Pulsar protocol or clients. The Rest API supports both non-partitioned and partitioned topics as well as basic and Avro base struct schema. 

Example use cases include:

• Send data to Pulsar from any frontend application built in any language
• Integrate Pulsar with existing automation tools
• Ingest Pulsar data into corporate dashboards and monitoring systems
• Provide instant access to data in motion for data scientist notebooks
• Ingest messages into a stream processing framework that may not support Pulsar

Certified as Red Hat OpenShift Operators

StreamNative Private Cloud is already certified as a Red Hat OpenShift Operator. The certifications of StreamNative Private Cloud on OpenShift brings three key benefits to StreamNative customers:

• Enterprise-grade security and reliability: Organizations with strict security protocols can confidently use the operators to run Pulsar on OpenShift knowing the operators meet Red Hat’s standards of security and reliability.
• Easy installation: Available in the Red Hat Ecosystem Catalog, you can easily install Pulsar Operators in the OpenShift GUI at the click of a button.
• Automated operator upgrades: You can automate upgrades for the operators through OpenShift without requiring extra effort to execute the upgrade.

What’s next

 StreamNative Private Cloud is in rapid iteration. Our next key roadmap items include: 

• Integrate with API Keys: API keys offer both a flexible authentication solution that can work with any client, and a revokable key that can be rotated on a regular interval for security and compliance, or immediately revoked in the event of a security incident.
• Integrate with pfSQL: pfSQL is a lightweight SQL-like tool that simplifies real-time data processing built on top of Pulsar functions.‍
• Full support for Kafka on StreamNative (KSN): KSN is an enterprise solution supports KStreams, KSQL, KTables with Topic Compaction, Schema Registry for the Java Client, and Kerberos Authentication for Kafka Clients.

Conclusion

With StreamNative Private Cloud, it's easier than ever to manage your Pulsar Cluster, and you'll get more functionality than ever before.

Sign up now for a trial license to try StreamNative Private Cloud for yourself. For more detailed guidance and insights, check out the Private Cloud docs.