Building Connectors On Pulsar Made Simple

Why Apache Pulsar Connectors?

Pulsar connectors enable Pulsar to quickly and easily integrate with various external systems. In fact, according to the 2021 Pulsar User Survey Report (which will be published later this month), connectors are one of the most-used Pulsar features with 30% of Pulsar users using connectors.

To facilitate connector development and improve their ease of use, we launched StreamNative Hub in 2020 to provide a single place to find, download, use, store, and share Pulsar related extensions, and offer a broad spectrum of Pulsar integrations. Since its launch last year, dozens of connectors have been created and added to the Hub. Some popular Pulsar plugins on StreamNative Hub include AWS SQS connector, AWS Lambda connector, AMQP1_0 connector, IoTDB connector, and more.

In this blog, we introduce recent updates that make developing and using Pulsar connectors even easier.

About StreamNative Hub

StreamNative Hub is an app store for developing event streaming applications and provides dozens of plugins and integrations. Its key components include:

• Connectors: Allow you to move streaming data in and out of Pulsar, which simplifies integration for enterprises bringing Pulsar into their existing infrastructure. All Pulsar built-in connectors are shipped in the StreamNative Hub.
• Offloader: Allow you to offload the majority of the data from BookKeeper to external remote storage, which provides a cheaper form of storage that readily scales with the volume of data.
• Protocol handler: Allow you to support other messaging protocols natively and dynamically in Pulsar brokers on runtime, which streamlines operations with Pulsar’s enterprise-grade features without modifying code. Kafka, AMQP, and MQTT are supported.

As more and more members have contributed and used connectors, we’ve identified some opportunities to improve the Hub’s ease of use, read on to learn more.

New Pulsar Connector Development Guide

To simplify the integration between Pulsar and external systems, we created a new development guide, Pulsar Connector Development Guide, that developers can reference to improve productivity and boost efficiency when developing a connector. This guide helps with the following:

• Developing a New Connector
• If you need to pipe data in or out of Pulsar and other systems that do not have a connector yet, you can read the Pulsar Connector Development Guide. It contains step-by-step guidelines for how to develop and contribute a connector to StreamNative Hub, including detailed instructions and various templates for both code and documentation.
• Promoting Awareness and Usage of an Existing Connector
• If you already developed a connector and want to make it available to the community, we recommend you host it in a public repository and show it on StreamNative Hub. You can host the connector repo at your desired location and then sync the documentation to StreamNative Hub using a simple script with just one line of code by following the instructions in the Pulsar Connector Development Guide.

Future StreamNative Hub Upgrades

We are continuously looking for new ways to improve StreamNative Hub and we are working on additional upgrades, such as adding more comprehensive tests to improve the usability, reliability, and performance of connectors. You can also expect more connectors to be deployed and adopted on more cloud providers with GUI tools. Stay tuned!

Contribute Your Connector

If you develop connectors, we encourage you to add your connector to StreamNative Hub! In StreamNative Hub, your connector will get exposure to the widest possible audience and enjoy faster innovation cycles of development. You will also be contributing to a robust Pulsar ecosystem.

Get Involved in the Pulsar Community

In addition to adding a connector, there are more ways you can contribute, including:

• Improve documentation!
• The documentation hosted at StreamNative Hub is open source. Feel free to submit or request changes (fix typos, add clarifications, and more).
• Report bugs.
• Review pull requests.
• Provide feedback on proposed features, enhancements, or designs.
• Suggest new features.
• Answer questions in issues or channels.

Ready to Get Started?

Start your journey with connectors now with the Quick Start Guide!

Happy Connectoring!