An Operator’s Guide: Configuring Geo-Replication with the Pulsar Resources Operator

Introduction

Apache Pulsar is a messaging and streaming platform that provides various features for businesses to manage their data and ensure its availability and reliability. One of the most powerful features of Pulsar is its built-in geo-replication functionality, which enables businesses to replicate data across multiple data centers, ensuring that data remains accessible and dependable even in the face of network failures or degraded performance between data centers.

By following best practices, such as monitoring the replication backlog, planning for capacity, and throttling replication, businesses can maximize the benefits of geo-replication in Pulsar and maintain a high-performing and dependable messaging system.

While configuring geo-replication using CLI tools such as pulsar-admin and pulsarctl is possible, there are better practices than this when running a Pulsar cluster on Kubernetes. These methods can be complex and time-consuming, requiring in-depth knowledge of the Pulsar ecosystem. To make this process more straightforward and efficient, the Pulsar Resources Operator provides a streamlined approach to configuring and managing Pulsar resources on a Kubernetes cluster. The Pulsar Resources Operator is a controller that automates the management of Pulsar resources through manifest files, enabling full lifecycle management for the resources necessary for geo-replication, including creation, update, and deletion for the following resources:

• PulsarConnection
• Tenants
• Namespaces
• Topics
• Permissions

With the Pulsar Resources Operator, configuring and managing multi-cluster environments no longer requires expertise in command-line tools. In the latest release of the Pulsar Resources Operator, version 0.3.0, we introduced a new Custom Resource named PulsarGeoReplication. This controller provides a declarative way to configure and manage geo-replication efficiently. 

Get started with the Pulsar Resources Operator

You can install the Pulsar Resources Operator using the officially supported pulsar-resources-operator Helm chart. It provides Custom Resource Definitions (CRDs) and Controllers to manage Pulsar resources.

Prerequisites

• Install kubectl (v1.16 - v1.25), compatible with your cluster (+/- 1 minor release from your cluster).
• Install Helm (v3.0.2 or higher).
• Prepare a Kubernetes cluster (v1.16 - v1.25).
• Prepare a Pulsar cluster.

Steps

To install the Pulsar Resources Operator, perform the following steps.

1. Add the StreamNative chart repository.

2. Install the operator using the pulsar-resources-operator Helm chart.

Configuring Geo-Replication using the Pulsar Resources Operator

We added a new Custom Resource PulsarGeoReplication, which describes the unidirectional replication between Pulsar clusters. Below, the `PulsarGeoReplication` CR shows the replication from the us-east-cluster to its destination cluster. 

We added a new field, `geoReplicationRefs`, to the `PulsarTenant`, `PulsarNamespace` and `PulsarTopic` resources to enable geo-replication at the tenant, namespace or topic level. However, note enabling geo-replication at the topic level requires an additional step of enabling the topic-level-policy on the Pulsar cluster.

Here we use two Pulsar clusters, `us-east` and `us-west`, running on two separate Kubernetes platforms, to demonstrate the geo-replication setup based on the Pulsar Resources Operator.

On the `us-east` cluster, we create a `PulsarConnection` for the local cluster and the destination cluster to establish a connection between the two clusters.

Once the `PulsarConnection` is established, we use the `PulsarGeoReplication` Custom Resource to configure the replication from `us-east` to `us-west`.

We can now create the `PulsarTenant`, `PulsarNamespace` and `PulsarTopic` resources with the `geoReplicationRefs` configuration, which enables geo-replication between the source and destination clusters.

On the `us-west` cluster, we need to create the corresponding resources in reverse as follows. We then create `PulsarTenant`, `PulsarNamespace`, and `PulsarTopic` resources with the `geoReplicationRefs` configuration, as before, but with the source and destination clusters reversed.

This sets up a unidirectional replication of messages from the `us-west` cluster to the `us-east` cluster, providing a complete geo-replication solution between the two clusters.

Tips and best practices for geo-replication

After setting up the geo-replication through the Pulsar Resources Operator, you can manage and monitor the replication to ensure it’s working properly. Here are some tips and best practices to help you do that.

‍Modifying resource configuration

The Resource Operator provides a declarative API to manage the resource objects, which means you can modify your Custom Resource YAML configuration and apply the changes using `kubectl apply`.  

However, it’s important to be aware of the implicit dependencies between different resource objects when modifying the configuration.

• `PulsarGeoReplication` CR depends on the `PulsarConnection` object
• `PulsarTenant`, `PulsarNamespace`, `PulsarTopic` CR depends on the `PulsarConnection` and `PulsarGeoReplication` objects.
• `PulsarNamespace`, `PulsarTopic` CR with `geoReplicationRefs` field depends on the target `PulsarTenant` with `geoReplicationRefs`. 

Monitoring replication status

To monitor the status of the geo-replication, you can use the Pulsar Broker Replication metrics. These metrics allow you to track the performance and health of the replication, with detailed information on the replication status, including the number of messages replicated, the lag between the primary and secondary clusters, and the replication rate.

Upgrading from 0.1.x or 0.2.x Resources Operator

If you have existing `PulsarConnection` objects running with the 0.1.x or 0.2.x Resource Operator and want to configure the geo-replication, you need to add the `brokerServiceURL` and `clusterName` fields for the `PulsarConnection` objects after upgrading to 0.3.0 version. These fields are required for geo-replication to work properly.

Integrating with GitOps for version control and approval

You can integrate the Pulsar Resources Operator with GitOps workflows for version control and approval of your Kubernetes configurations. With tools like ArgoCD and GitHub, you can control Pulsar CR changes in an elegant way. This integration ensures that all changes to the Pulsar cluster are properly tracked, reviewed, and deployed in a controlled manner. Check out Managing Pulsar CRs with ArgoCD to learn more.

Cleaning up geo-replication

If you want to clean up geo-replication on resource objects, it’s best to follow these steps: 

1. Remove the `geoReplicationRefs` field on the `PulsarTenant`, `PulsarNamespace`, `PulsarTopic` CRs. 
2. Delete the `PulsarGeoReplication` object. 
3. Delete the destination `PulsarConnection` object.

This will ensure that all geo-replication resources are properly removed and that there are no lingering dependencies.

Testing failover

It's important to test your geo-replication failover setup regularly to ensure that it works as expected in case of a disaster. You can do this by simulating a network or hardware failure on one of your Pulsar clusters and verifying that the replication fails over to the other cluster seamlessly.

By following these tips and best practices, you can ensure that your geo-replication setup is reliable, efficient, and easy to manage.

Conclusion

StreamNative’s Pulsar Resources Operator provide a reliable and efficient way to set up geo-replication for disaster recovery. With geo-replication, data can be replicated across multiple clusters in different geographical locations. In this guide, we have covered how to set up geo-replication using the Pulsar Resources Operator. We have also discussed some tips and best practices to follow when configuring geo-replication on resource objects.

StreamNative offers a comprehensive solution for managing Pulsar clusters on Kubernetes. In addition to the Pulsar Resources Operator, StreamNative provides a set of Kubernetes controllers called Pulsar Operators that simplify the deployment and management of Pulsar clusters on Kubernetes. With StreamNative Pulsar Operators, SREs can bridge the gap between complex operations of distributed systems and the declarative nature of GitOps, allowing them to deploy changes with confidence.

More Resources

Join the Apache Pulsar community today and take part in shaping the future of messaging and streaming. Check out the GitHub repos, and documentation, and contribute to building an exciting project. 

• [Blog] Introducing Pulsar Resources Operator for Kubernetes
• [Blog] Pulsar Operators Tutorial Part 1: Create an Apache Pulsar Cluster on Kubernetes
• [Blog] Pulsar Operators Tutorial Part 2: Manage Pulsar Custom Resources with ArgoCD
• [Blog] Pulsar Operators Tutorial Part 3: Create and Deploy a Containerized Pulsar Client
• [Blog] Pulsar Operators Tutorial Part 4: Use kpack to Streamline the Build Process

‍