Automated Geo-Replication set up in StreamNative Cloud Pulsar instances

Geo-replication has always been a standout feature at the core of Apache Pulsar, enabling seamless data replication across geographically distributed regions for enhanced reliability and availability. However, setting up Geo-replication has traditionally required additional effort, including configuring a global configuration store, ensuring network connectivity, and managing complex credentials. At StreamNative, we’ve worked hard to address these challenges and simplify the process for our customers. Today, we’re excited to introduce Automated Geo-Replication setup in StreamNative Cloud Pulsar instance, a new feature available across all StreamNative product lines that streamlines the setup of Geo-replication and brings enterprise-grade multi-cluster functionality to all our users.

What is Apache Pulsar Geo-Replication?

‍

Apache Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For example, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar's geo-replication mechanism, messages can be produced and consumed in different geo-locations. 

The Challenge of Traditional Geo-Replication Setup

Setting up the geo-replication in the Apache Pulsar requires some expertise and manual steps like selecting the geo-replication solution between global configuration store or separate configuration store, network infrastructure connectivity, cluster authentication credentials etc. For existing StreamNative Clusters, enabling replication between instances required additional administrative overhead. Customers opting for our "Pro" tier could rely on the StreamNative team to configure Geo-replication during provisioning, but this approach wasn’t available to all customers or products.

A Simpler Solution: AutomatedGeo-Replication setup in StreamNative Cloud

To make Geo-replication accessible to all our customers and eliminate the need for complex setup, We introduced the automated multi-cluster creation in a StreamNative Cloud Pulsar instance. The Pulsar instance on StreamNative Cloud is a group of Pulsar clusters that function together as a unified entity. This enhancement allows users to create multiple Pulsar clusters within a single instance effortlessly, ensuring geo-replication is configured with just a few clicks, without the manual configuration traditionally required.

Managing multi-clusters on StreamNative Cloud

StreamNative Cloud Console now supports creating multiple Pulsar clusters under the same Pulsar Instance, and you can create the second or following Pulsar clusters on the Pulsar Instance page. 

You can manage the message replication on the Pulsar tenants and namespaces with flexibility. On the namespace configuration, you can select the replication clusters from the dropdown and  any topics that producers or consumers create within that namespace are replicated across clusters. 

Replicated subscription is useful within geo-replication, you can in case of failover, a consumer can restart consuming from the failure point in a different cluster: 

Consumer<String> consumer = client.newConsumer(Schema.STRING)
          .topic("my-topic")
          .subscriptionName("my-subscription")
          .replicateSubscriptionState(true)
          .subscribe();

Conclusion

StreamNative Cloud's support for Pulsar instance geo-replication is a powerful feature that simplifies the data resilience and availability across global regions. For more information on Automated Geo-Replication setup, please visit the documentation. 

‍