Pulsar connectors enable Pulsar to quickly and easily integrate with various external systems. In fact, according to the 2021 Pulsar User Survey Report (which will be published later this month), connectors are one of the most-used Pulsar features with 30% of Pulsar users using connectors.
To facilitate connector development and improve their ease of use, we launched StreamNative Hub in 2020 to provide a single place to find, download, use, store, and share Pulsar related extensions, and offer a broad spectrum of Pulsar integrations. Since its launch last year, dozens of connectors have been created and added to the Hub. Some popular Pulsar plugins on StreamNative Hub include AWS SQS connector, AWS Lambda connector, AMQP1_0 connector, IoTDB connector, and more.
In this blog, we introduce recent updates that make developing and using Pulsar connectors even easier.
About StreamNative Hub
StreamNative Hub is an app store for developing event streaming applications and provides dozens of plugins and integrations. Its key components include:
Connectors: Allow you to move streaming data in and out of Pulsar, which simplifies integration for enterprises bringing Pulsar into their existing infrastructure. All Pulsar built-in connectors are shipped in the StreamNative Hub.
Offloader: Allow you to offload the majority of the data from BookKeeper to external remote storage, which provides a cheaper form of storage that readily scales with the volume of data.
Protocol handler: Allow you to support other messaging protocols natively and dynamically in Pulsar brokers on runtime, which streamlines operations with Pulsar’s enterprise-grade features without modifying code. Kafka, AMQP, and MQTT are supported.
As more and more members have contributed and used connectors, we’ve identified some opportunities to improve the Hub’s ease of use, read on to learn more.
New Pulsar Connector Development Guide
To simplify the integration between Pulsar and external systems, we created a new development guide, Pulsar Connector Development Guide, that developers can reference to improve productivity and boost efficiency when developing a connector. This guide helps with the following:
Developing a New Connector
If you need to pipe data in or out of Pulsar and other systems that do not have a connector yet, you can read the Pulsar Connector Development Guide. It contains step-by-step guidelines for how to develop and contribute a connector to StreamNative Hub, including detailed instructions and various templates for both code and documentation.
Promoting Awareness and Usage of an Existing Connector
If you already developed a connector and want to make it available to the community, we recommend you host it in a public repository and show it on StreamNative Hub. You can host the connector repo at your desired location and then sync the documentation to StreamNative Hub using a simple script with just one line of code by following the instructions in the Pulsar Connector Development Guide.
Future StreamNative Hub Upgrades
We are continuously looking for new ways to improve StreamNative Hub and we are working on additional upgrades, such as adding more comprehensive tests to improve the usability, reliability, and performance of connectors. You can also expect more connectors to be deployed and adopted on more cloud providers with GUI tools. Stay tuned!
Contribute Your Connector
If you develop connectors, we encourage you to add your connector to StreamNative Hub! In StreamNative Hub, your connector will get exposure to the widest possible audience and enjoy faster innovation cycles of development. You will also be contributing to a robust Pulsar ecosystem.
Get Involved in the Pulsar Community
In addition to adding a connector, there are more ways you can contribute, including:
The documentation hosted at StreamNative Hub is open source. Feel free to submit or request changes (fix typos, add clarifications, and more).
Review pull requests.
Provide feedback on proposed features, enhancements, or designs.
Guangning E is an Apache Pulsar committer and the main contributor to Apache Pulsar IO and Apache Pulsar Manager. He works as a senior software engineer at StreamNative, where he specializes in cloud platform, cloud computing, and big data related fields.
Yu Liu is an Apache Pulsar PMC member and a content strategist from StreamNative.