Announcing: The Pulsar PMC Published The 2020 Apache Pulsar User Survey Report!

Overview
Get started
Install and upgrade
Overview
Bare metal
Kubernetes (Helm)
Configure
Secure
Manage and monitor
Connect
Process
Release notes

Overview

You can install and run StreamNative Platform on on-premises environments with a ZIP/TAR archive, or a DEB/RPM package.

You can also automate deployment to Kubernetes on a public or private cloud using Helm.

This document outlines the general guidelines of deploying a production-ready multi-node StreamNative Platform cluster. It applies to both On-Premises deployments and Kubernetes deployments.

Guidelines

Components

StreamNative Platform deployment is mainly comprised of two parts:

  • The first part is an Pulsar cluster, which includes all the Pulsar open-source components, and StreamNative community and commercial components.

  • The second part is StreamNative Control Center, which uses Prometheus for collecting metrics, Grafana for visualizing metrics in dashboards, Alertmanager for handling alerts, and Pulsar Manager for managing and operating Pulsar clusters and resources.

The figure below outlines a typical deployment of a StreamNative Platform deployment. The StreamNative Control Center does not have any tight dependencies on an Apache Pulsar cluster. A StreamNative Control Center can be used for monitoring, managing, and operating multiple Apache Pulsar clusters.

StreamNative Platform Components

ServiceUrl

Pulsar clients use a ServiceUrl to connect to an Apache Pulsar cluster to produce and consume messages, as well as manage and operate Pulsar resources (such as tenants, namespace, topics, etc). So before deploying an Apache Pulsar cluster, you need to plan how to expose the Pulsar cluster to your applications for their usage. There are two approaches:

  • Expose brokers directly
  • Use proxies

Expose brokers directly

If you expose brokers directly, the Pulsar applications should run in the same network environment with your brokers. Because Pulsar clients are required to establish TCP connections directly with each of the brokers.

The figure below illustrates a typical setup of an Pulsar cluster exposing brokers directly to Pulsar applications.

StreamNative Platform Components

Use proxy

If you plan to run a Pulsar cluster in a network environment (such as Kubernetes) where the Pulsar clients can not establish TCP connections directly with each of the brokers. You need to set up a set of Pulsar proxies in front of the brokers. So the Pulsar proxies can look up the ownerships of topics and establish the direct TCP connections with the right owner brokers.

The figure below illustrates a typical setup of an Pulsar cluster exposing the service to Pulsar applications using Proxy.

StreamNative Platform Components

In both cases, you can configure a DNS and a load balancer for your brokers or proxies. So the Pulsar applications can use one single domain to connect to the cluster. If it is non-trivial for you to set up a DNS or a load balancer, you can configure your Pulsar applications to access the cluster using a Multi-Host service URL comprised of a list of bootstrap brokers (or proxies) (for example, pulsar://broker1:6650,broker2:6650,broker3:6650).

Deploy sequence

There is a certain sequence when deploying a Pulsar cluster. The sequence is highlighted with red numbers in the above figures and described as below.

  1. Deploy ZooKeeper. Make sure the ZooKeeper cluster is up running healthy before moving to the next step.
  2. Initialize Pulsar cluster metadata using bin/pulsar initialize-cluster-metadata. The tool initializes the metadata for both Pulsar and BookKeeper and creates a cluster configuration with its service URLs.
    • Run this tool from a machine that is able to connect to ZooKeeper server and configuration store (if there is a separate one).
    • Make sure the cluster name is correct and it is the cluster name that you use for configuring brokers.
  3. Deploy bookies. Make sure at least an ensemble of bookies are up running healthy before moving to the next step.
  4. Deploy brokers.
  5. Deploy proxies (optional).
  • For more information about deploying StreamNative Platform on on-premises environment, see the instructions below.
  • For more information about deploying StreamNative Platform on Kubernetes, see Helm Chart.

On-premises deployment

Kubernetes deployment