Unveiling StreamNative’s Enhanced Connector Experience

Today, with data spread far and wide across myriad systems, the seamless movement of this data between diverse systems and platforms has become paramount. Pulsar IO Connectors have been a bridge facilitating this critical data transfer, ensuring that every byte finds its rightful place whether you’re moving data into Pulsar or channeling it outwards into other systems. That’s why we’re excited to announce an enhanced IO Connector experience, exclusively on StreamNative Cloud. These updates will not only streamline the process but also significantly reduce development efforts, making the sometimes onerous task of data migration between systems as smooth as possible.

Custom Connectors: Generally Available on all new Hosted and BYOC Pulsar Clusters on StreamNative Cloud

While the list of connectors available on StreamNative Cloud is ever-growing, there will always be a need to connect a new system to your Pulsar Cluster, or to connect an existing system to Pulsar in a way the current connector offerings don’t support. That realization convinced us that we need to do two things at once: not only add more StreamNative-build IO Connectors, but also give customers a way to build their own. To that end, we’re excited to announce that Custom Connectors are now generally available on StreamNative Cloud for all new Hosted and Bring Your Own Cloud (BYOC) Clusters!

Built atop Pulsar Functions, Custom Connectors give you the power to develop bespoke Connectors tailored to your unique data needs. With the ability to design connectors from the ground up, we're ensuring that you will never be limited to the existing IO Connectors we offer (though these are pretty good too - check them out here). All you need to do is implement the Source or Sink interface, and you’ve got your own Connector! To learn more Custom Connectors, check out the documentation that walks users through creating, testing, and deploying a custom connector.

IO Connector Debugging and Management: Generally Available on All StreamNative Clusters

As the gates that control data flowing into and out of your Pulsar Cluster, Connectors are a fundamentally important part of your Cluster. So it’s critical to have insight into what's going on with them. We understand this, and have shipped a number of improvements to the IO Connector management and debugging experience:

• Users can now easily access logs and exceptions directly from the StreamNative Console, providing a clearer window into IO Connector operations and streamlining the debugging process. Just navigate to an individual Connector’s page from the Connectors page in the Console, and click on the logs and exceptions tabs.
• Additionally, IO Connector logs are now seamlessly routed to a Pulsar topic through sidecar, enabling programmatic consumption, so if you want to integrate your IO Connector logs into your logging and alerting solutions, you can now do so.

Terraform and pulsarctl Support for Connectors: Generally Available for new Pulsar Clusters on StreamNative Cloud

For users who want to manage their Pulsar Cluster programmatically, we’ve got good news: you can now submit and modify Source and Sink Connectors via both Terraform and pulsarctl. So if you want to manage your Pulsar Cluster solely via Terraform so that all your settings are checked into source control, or you prefer to to do via CLI instead of the Console, you can now do so! To learn more about using Terraform and pulsarctl with Connectors check out our docs on deploying Connectors.

Coming Soon to StreamNative Cloud: A Comprehensive Overhaul of the StreamNative Console Connector Experience

Setting up a Connector has always been an inherently laborious task, requiring users to track down various credentials and settings during the setup process, then testing whether the Connector works, and modifying it to optimize performance once setup. Our team has crafted a new experience that will redefine the Connector creation process, making it a breeze to set up new IO Connector on StreamNative Cloud. The new experience will guide you through setting up your connectors, with step-by-step instructions for each setting, and recommendations on configuration for maximum performance. Here’s a quick preview of it:

The future of IO Connectors on StreamNative Cloud

While we’re excited about the improvements we’ve made so far, this is merely the beginning of an exhilarating journey ahead. We’re going to continue and accelerate our work of onboarding more IO Connectors, and improving our existing Connectors. We envision a vibrant ecosystem of constantly improving connectors, and a rich experience across the StreamNative Console, CLI tools, and Terraform. Our goal is to give you a robust set of tools that work no matter your tech stack, that works so seamlessly you forget it’s even running.

If you’d like to try out our enhanced Connector experience for yourself, sign up for a StreamNative Hosted Cluster today, or contact our sales team if you’d like to try a StreamNative BYOC Cluster.

Stay tuned for updates, and as always, happy Streaming!

‍

‍