First-Ever Pulsar Summit Virtual Conference 2020 | June 17th and 18th

Overview
Get started
Install and upgrade
Configure
Secure
Manage and monitor
Connect
Process
Release notes

Overview

Download StreamNative Platform to get started.

Founded by the original developers of Apache Pulsar and Apache BookKeeper, StreamNative provides a cloud-native messaging and event-streaming platform, enabling you to build a real-time application and data infrastructure for both real-time and historical events.

Powered by Apache Pulsar, StreamNative Platform makes it easy to build mission-critical messaging and streaming applications and real-time data pipelines by integrating data from multiple sources into a single, central messaging and event streaming platform for your company. StreamNative Platform lets you focus on how to maximize business value from real-time data rather than worrying about the underlying mechanisms such as how data is messaged between various systems and how data is stored reliably for processing.

Specifically, StreamNative Platform simplifies:

  • Publishing-and-consuming messages using popular messaging protocols (including Apache Pulsar and Apache Kafka)
  • Connecting various data sources to Pulsar
  • Building real-time applications with Pulsar
  • Integrating your data processing infrastructure with Pulsar
  • Securing, monitoring, and managing your Pulsar deployments

Out of the box, the latest features of Pulsar and additional StreamNative features are shipped in each StreamNative Platform release. Currently, the latest StreamNative Platform version is 1.0.0. Below are the compatibilities of the StreamNative Platform and each component.

Component Version
Core Apache Pulsar 2.5.0-ad0224407
Kafka-on-Pulsar (KoP) 0.2.0
Pulsar Flink Connector 2.4.8
Pulsar Spark Connector 2.4.5
Control Center Manage
Pulsar Manager 0.3.0
pulsarctl 0.4.0
Monitor
Prometheus 2.15.2
Node exporter 0.18.1
Alertmanager 0.20.0
Grafana 5.3.4

Apache Pulsar

At the core of StreamNative Platform is Apache Pulsar, the next-generation cloud-native messaging and streaming platform.

The key capabilities of Apache Pulsar are:

Apache Pulsar has received global adoption from major technology companies such as Verizon Media, Narvar, Overstock, Nutanix, Yahoo! JAPAN, Tencent, OVHCloud, and Clever Cloud, who rely on its ability to deliver on performance, scalability, and resiliency. Pulsar usage has largely been driven by the market’s increased demand for real-time, data-enabled technologies, from asynchronous applications to core business applications to ETL.

The key components of the Pulsar open source project are:

  • Brokers

    Brokers form the messaging and serving tier for Pulsar.

  • Bookies

    Bookies form the storage tier for Pulsar.

  • Functions Workers

    Functions Workers form the serverless processing tier for Pulsar.

  • Proxy

    Pulsar proxy is an optional gateway that you can run in front of the brokers in a Pulsar cluster.

  • ZooKeeper

    Pulsar relies on ZooKeeper for a wide variety of configuration-related and coordination-related tasks.

  • Client APIs

    • Producer API: allows an application to publish a stream of events to one or more Pulsar topics.
    • Consumer API: allows an application to subscribe to one or more topics and receive the stream of events produced to them.
    • Functions API: allows an application to create, deploy, and manage serverless functions that consume events from one or more Pulsar topics, apply a user-supplied processing logic to each event, and publish the results of the computation to another topic.
    • Connector API: allows an application to create, deploy, and manage connectors that interact with external systems, such as Apache Cassandra, Aerospike, and many others.
    • Admin API: allows an application to manage Pulsar resources (for example, namespaces, topics, functions, connectors, and so on) in a programmable way.

StreamNative features

Each release of StreamNative Platform includes the latest release of Pulsar and additional tools and services that make it easier to build and manage a cloud-native messaging and event-streaming platform. StreamNative Platform delivers both community and commercially licensed features that complement and enhance your Pulsar deployment.

Connect

Kafka on Pulsar

KoP (Kafka on Pulsar) is a protocol handler that brings native Kafka protocol support to Apache Pulsar. It is implemented as a Pulsar protocol handler with protocol name - kafka and it can be easily installed to your existing Pulsar brokers.

By adding the Kafka-on-Pulsar protocol handler to your existing Pulsar brokers, you can now migrate your existing Kafka applications and services to Pulsar without modifying the code. This enables Kafka applications to leverage Pulsar’s powerful features, such as:

  • Streamlined operations with enterprise-grade multi-tenancy
  • Simplified operations with a rebalance-free architecture
  • Infinite event stream retention with Apache BookKeeper and tiered storage
  • Serverless event processing with Pulsar Functions

For more information, see KoP.

Process

Apache Fink is a next-generation stream processing system at scale. It provides almost every important mechanism for building a modern sophisticated streaming application, such as backpressure, user-defined state, and various streaming windows.

Pulsar Flink Connector integrates Apache Flink to query and process real-time event streams stored in Pulsar.

For more information, see Pulsar Flink Connector.

Pulsar Spark Connector

Apache Spark is used for a wide variety of jobs, ranging from ETL, machine learning, to stream processing. It is designed for fast and general data processing at a large data scale and has a wide adoption as a "one-stop" big data solution.

Pulsar Spark Connector integrates Apache Spark for querying and processing events stored in Pulsar.

For more information, see Pulsar Spark Connector.

Manage

Pulsar Manager

Pulsar Manager is a web-based GUI management and monitoring tool, to manage tenants, namespaces, topics, subscriptions, brokers, and clusters in your Pulsar deployments. It also supports dynamic configurations of multiple environments.

Pulsar Manager supersedes Pulsar Dashboard by providing an easy way for operators to manage Pulsar deployments, such as adding, deleting and updating tenants, namespaces, topics, and so on. It also collects and displays information of Pulsar, such as showing statistics of tenants, namespaces, topics, subscriptions, and so on.

For more information, see Pulsar Manager.

pulsarctl

pulsarctl, a command-line tool developed in Go using Pulsar Admin REST API, manages and operators Pulsar clusters and resources (such as tenants, namespaces, topics, schemas, sources, sinks, functions, etc).

pulsarctl is a better alternative tool of pulsar-admin. For the comparison between pulsarctl and pulsar-admin, see here.

For more information, see pulsarctl.

Monitor

StreamNative Platform integrates the following tools to monitor your Pulsar deployments.

  • Node exporter: collect and monitor the system statistics of the machines that host your Pulsar deployments
  • Prometheus: collect and monitor the metrics of each component of your Pulsar deployments
  • Alertmanager: define and handle alerts based on the metrics of your Pulsar deployments
  • Grafana: visualize the metrics in dashboards

Document

Use these guides to develop and deploy your Pulsar applications on StreamNative Platform.

To help you locate your desired contents quickly and efficiently, based on your roles, it is recommended that you read the following guides: