Introducing StreamNative Operators for Apache Pulsar

Apache Pulsar is the most advanced data streaming technology available today, however, managing efficiently the different components of a Pulsar cluster can be a complex task. To ease this complexity, StreamNative released the StreamNative Operator for Apache Pulsar and offered it under a free community license, which allows enterprises to start working with open-source Pulsar without dealing with the initial complexity of operating it so that teams can focus on trying and start implementing applications on top of it. 

Running Apache Pulsar On Kubernetes

Apache Pulsar is the ideal choice for enterprises that want an open-source, elastic, and multi-tenant data streaming and messaging platform, allowing them to centralize data management and drastically reduce operation costs compared to having multiple clusters of older technologies such as Kafka or RabbitMQ. Well-designed for optimal functionality and flexibility in a containerized environment, its architecture is somehow sophisticated as it ties together multiple open-source projects, in particular Apache BookKeeper for its storage component, and Apache Zookeeper for the management of distributed metadata. 

To ease this complexity, StreamNative - the company founded by the original creators of Apache Pulsar - released its battle-tested Kubernetes operators for Apache Pulsar, which embody StreamNative's accumulated expertise by the creators of Apache Pulsar and years of experience managing Pulsar clusters in a Kubernetes environment for large-scale companies like Verizon, Discord, or Iterable.

This set of Kubernetes operators makes it easy to deploy, manage, and scale Apache Pulsar clusters. They provide a declarative API that allows teams to define the Pulsar cluster configuration in Kubernetes manifests, and automatically manage the lifecycle of Pulsar brokers, proxies, and BookKeeper bookies.

The StreamNative Pulsar Operators offer a number of benefits, including:

• Simplified deployment: The operators automate the deployment of Pulsar clusters, so teams don't have to worry about manually configuring and managing the individual components.
• High Availability: StreamNative Pulsar Operators set up clusters in a highly available manner by default. They manage replica placement, broker distribution, and failover mechanisms, ensuring that event streams stay reliable even in the face of failures.
• Declarative configuration: The operators use a declarative API, so teams can define the Pulsar cluster configuration in Kubernetes manifests. This makes it easy to manage the Pulsar cluster and to roll back changes if necessary.

Protocol ecosystems

StreamNative Pulsar Operators support pillars of the StreamNative protocol ecosystem: Kafka on Pulsar (KoP), MQTT on Pulsar (MoP), and AMQP on Pulsar (AoP). These projects extend the capabilities of Apache Pulsar by providing compatibility with popular messaging and streaming protocols, offering organizations the flexibility to choose the best fit for their event-driven architectures.

With StreamNative Pulsar Operators, teams can leverage KoP, MoP, and AoP to empower organizations to embrace event-driven architectures to open up new avenues for innovation in IoT, real-time communication, and enterprise integration. 

Automated operations

Like all stateful applications, a BookKeeper cluster on Kubernetes is challenging to manage, but StreamNative Pulsar Operators offer several enhancements to facilitate the bookies' operation:

• Auto disable autorecovery: when upgrading the bookie’s version or changing the bookie configurations, the operator will automatically disable the autorecovery service in the cluster to avoid unexpected ledger replication and enable the autorecovery back after the upgrade ends. 
• Auto decommission job: Before removing a bookie pod in the cluster, the StreamNative Pulsar Operator will automatically trigger a decommissioning job to start the ledger replication, avoiding the risk of data loss. 
• Auto JVM configuration: The operator supports generating the JVM configuration based on the current Pod size with StreamNative Pulsar experts’ best practice.

OLM and OpenShift support

StreamNative Pulsar Operators support the Operator Lifecycle Manager to provide a simple declarative way to install, manage, and upgrade operators on a cluster.

Also, these operators are certified as Red Hat OpenShift Operators, which benefits:

• Enterprise-grade security and reliability: Organizations with strict security protocols can confidently use the operators to run Pulsar on OpenShift, knowing the operators meet Red Hat’s standards of security and reliability.
• Easy installation: Available in the Red Hat Ecosystem Catalog, you can easily install Pulsar Operators in the OpenShift GUI at the click of a button.
• Automated operator upgrades: You can automate upgrades for the operators through OpenShift without requiring extra effort to execute the upgrade.

Cloud-native networking

Istio is a popular cloud-native networking platform, and StreamNative Pulsar Operators integrate with Istio to bring a better cloud-native experience for Pulsar users by:

• Support creating and managing Istio VirualService and Gateway resources for Pulsar
• Support exposing Pulsar Protocol, Kafka Protocol, and MQTT Protocol through the Istio Gateway in a unified way.
• Leverage the Istio to provide encrypted traffic communication between Pulsar components

Getting started with StreamNative Pulsar Operators

A few lines are enough to set and run an operational Pulsar cluster.

Using the Helm to start the StreamNative Pulsar Operators: 

helm repo add streamnative https://charts.streamnative.io
helm repo update
helm upgrade --install pulsar-operator streamnative/pulsar-operator

Using below commands to provision a Pulsar cluster:

kubectl create ns pulsar
kubectl apply -f https://raw.githubusercontent.com/streamnative/charts/master/examples/pulsar-operators/quick-start.yaml

Look at our documentation for more details.

Empowering Enterprises with Advanced Streaming Technology

These tools are part of StreamNative's commitment to fostering a robust ecosystem for Apache Pulsar. By simplifying the deployment and management of Pulsar in an enterprise environment, we enable teams across your organization to benefit from this cutting-edge technology. Apache Pulsar offers superior messaging and streaming capabilities, and with the new StreamNative Kubernetes operator, it's more accessible than ever.

As data is increasingly becoming the backbone of successful decision-making and innovation, the value of an advanced, easy-to-manage data streaming technology cannot be overstated. We believe that our contribution will open the doors to new possibilities for businesses seeking to leverage real-time data. It is a significant step forward in making Apache Pulsar the go-to solution for enterprise data streaming needs.

The StreamNative Kubernetes operators for Apache Pulsar are available under a free community license. We invite you to explore how these tools can enhance your data streaming capabilities and help you unlock the full potential of Apache Pulsar in your organization.

StreamNative Cloud

Some enterprises would want access to Pulsar experts, 24/7 support, and fully managed Pulsar clusters. For those StreamNative proposes StreamNative Cloud, a fully managed, enterprise-grade version of Pulsar with additional features such as:

• Enhanced compatibility with the Kafka protocol
• Battle-tested Pulsar Functions
• Audit Logs
• Health detector
• Additional Security & Authentication features
• Managed connectors (sources & sinks)
• In-house console and CLIs to manage your clusters

StreamNative Cloud can be deployed on-premise, fully managed on your public cloud account, or used in SaaS mode. StreamNative has already helped dozens of engineering teams worldwide make the move to Pulsar, contact us for more information.