Founded by the original developers of Apache Pulsar and Apache BookKeeper, StreamNative provides a cloud-native messaging and event-streaming platform, enabling you to build a real-time application and data infrastructure for both real-time and historical events.
Powered by Apache Pulsar, StreamNative Platform makes it easy to build mission-critical messaging and streaming applications and real-time data pipelines by integrating data from multiple sources into a single, central messaging and event streaming platform for your company. StreamNative Platform lets you focus on how to maximize business value from real-time data rather than worrying about the underlying mechanisms such as how data is messaged between various systems and how data is stored reliably for processing.
Specifically, StreamNative Platform simplifies:
Out of the box, the latest features of Pulsar and additional StreamNative features are shipped in each StreamNative Platform release. Currently, the latest StreamNative Platform version is 1.0.0. Below are the compatibilities of the StreamNative Platform and each component.
|Pulsar Flink Connector||2.4.8|
|Pulsar Spark Connector||2.4.5|
At the core of StreamNative Platform is Apache Pulsar, the next-generation cloud-native messaging and streaming platform.
The key capabilities of Apache Pulsar are:
Apache Pulsar has received global adoption from major technology companies such as Verizon Media, Narvar, Overstock, Nutanix, Yahoo! JAPAN, Tencent, OVHCloud, and Clever Cloud, who rely on its ability to deliver on performance, scalability, and resiliency. Pulsar usage has largely been driven by the market’s increased demand for real-time, data-enabled technologies, from asynchronous applications to core business applications to ETL.
The key components of the Pulsar open source project are:
Brokers form the messaging and serving tier for Pulsar.
Bookies form the storage tier for Pulsar.
Functions Workers form the serverless processing tier for Pulsar.
Pulsar proxy is an optional gateway that you can run in front of the brokers in a Pulsar cluster.
Pulsar relies on ZooKeeper for a wide variety of configuration-related and coordination-related tasks.
Each release of StreamNative Platform includes the latest release of Pulsar and additional tools and services that make it easier to build and manage a cloud-native messaging and event-streaming platform. StreamNative Platform delivers both community and commercially licensed features that complement and enhance your Pulsar deployment.
KoP (Kafka on Pulsar) is a protocol handler that brings native Kafka protocol support to
Apache Pulsar. It is implemented as a Pulsar protocol handler
with protocol name -
kafka and it can be easily installed to your existing Pulsar brokers.
By adding the Kafka-on-Pulsar protocol handler to your existing Pulsar brokers, you can now migrate your existing Kafka applications and services to Pulsar without modifying the code. This enables Kafka applications to leverage Pulsar’s powerful features, such as:
For more information, see KoP.
Apache Fink is a next-generation stream processing system at scale. It provides almost every important mechanism for building a modern sophisticated streaming application, such as backpressure, user-defined state, and various streaming windows.
Pulsar Flink Connector integrates Apache Flink to query and process real-time event streams stored in Pulsar.
For more information, see Pulsar Flink Connector.
Apache Spark is used for a wide variety of jobs, ranging from ETL, machine learning, to stream processing. It is designed for fast and general data processing at a large data scale and has a wide adoption as a "one-stop" big data solution.
Pulsar Spark Connector integrates Apache Spark for querying and processing events stored in Pulsar.
For more information, see Pulsar Spark Connector.
Pulsar Manager is a web-based GUI management and monitoring tool, to manage tenants, namespaces, topics, subscriptions, brokers, and clusters in your Pulsar deployments. It also supports dynamic configurations of multiple environments.
Pulsar Manager supersedes Pulsar Dashboard by providing an easy way for operators to manage Pulsar deployments, such as adding, deleting and updating tenants, namespaces, topics, and so on. It also collects and displays information of Pulsar, such as showing statistics of tenants, namespaces, topics, subscriptions, and so on.
For more information, see Pulsar Manager.
pulsarctl, a command-line tool developed in Go using Pulsar Admin REST API, manages and operators Pulsar clusters and resources (such as tenants, namespaces, topics, schemas, sources, sinks, functions, etc).
pulsarctl is a better alternative tool of pulsar-admin. For the comparison between pulsarctl and pulsar-admin, see here.
For more information, see pulsarctl.
StreamNative Platform integrates the following tools to monitor your Pulsar deployments.
Use these guides to develop and deploy your Pulsar applications on StreamNative Platform.
To help you locate your desired contents quickly and efficiently, based on your roles, it is recommended that you read the following guides: