KSN: Bringing Enterprise-Grade Kafka Support to StreamNative Pulsar Clusters

KSN is now part of the Ursa engine.

Apache Pulsar is the next generation of data streaming technology, bringing a slew of advantages over Apache Kafka, especially for large-scale enterprise applications, offering native multi-tenancy, geo-replication, tiered storage, and unmatched scalability and elasticity. However, switching technologies isn't a simple decision. Organizations that have heavily invested in Kafka need a smooth transition path if they are to consider a move to Pulsar.

At StreamNative, we've long recognized this migration challenge. Over the past few years, we've empowered Kafka developers with the ability to utilize Pulsar via our open source project, KoP, which embeds the Kafka protocol handler inside the Pulsar broker. With KoP, Kafka developers can immediately begin leveraging Pulsar's innovations while retaining the familiarity of Kafka.

While KoP marked a significant milestone, we still identified gaps that needed to be filled, especially to cater to the sophisticated needs of enterprise Kafka users — specifically, the need for supporting key Kafka features like KStreams or KSQL.

Introducing KSN

That's why we're excited to present KSN, a Kafka protocol compatible layer running on Pulsar 3.1, now in Public Preview, an offering tailored to enterprises using Kafka who want to leverage Pulsar's enhanced capabilities. KSN builds on KoP, but contains even more features:

• All of the Kafka features you're used to: KSN supports all the Kafka features you're accustomed to, including KStreams, KSQL, KTables with Topic Compaction, Schema Registry for the Java Client, and Kerberos Authentication for Kafka Clients.
• Great developer experience: KSN features streamlined local testing, with its own testcontainers module.
• Robust and reliable: Architecturally designed for scale and subjected to rigorous resilience testing. We've fortified KSN through comprehensive testing, ensuring it meets and exceeds the demands of large-scale Kafka operations.

What’s next for KSN (Coming Soon)

We’ve already started work on the next version of KSN, which will contain some exciting enhancements over and above what’s already in this version:

• Transaction support with Topic Compaction: KSN’s next version will support transactions with Topic Compaction, allowing for atomic writes and highly accurate stream processing.
• Role Based Access Control: KSN’s next version will feature role based access control, built upon Pulsar’s updated authorization model.
• Unified Schema Registry: KSN already implements a Kafka Schema Registry, but KSN’s next version will expand on this by featuring a unified schema registry that can be used by both Pulsar and Kafka clients interchangeably. In other words, Pulsar consumers can consume messages with a schema produced by Kafka producers, and vice versa, making the transition from Kafka to Pulsar on StreamNative even easier.

Take Your Streaming Capabilities to the Next Level

KSN is now available for our StreamNative Hosted and BYOC customers running Pulsar 3.1 or later. If you’re a current StreamNative customer and would like to try it out, just contact our support team. To learn more about KSN, check out our YouTube playlist, which walks you through getting started, and using Kafka features on StreamNative, such as KStreams, KSQL and using the Kafka Schema Registry.

Position yourself at the forefront of streaming technology with KSN. If you have questions or want to know more, our sales team is available to guide you - reach out today!

‍