How to Migrate from RabbitMQ to Apache Pulsar™

Note:

The AoP plugin supports only the 0-9-1 protocol with basic produce and consume functionalities, and does not include advanced features such as transactions. It is available as an open-source plugin and is only offered as a private preview feature in the Private Cloud distribution. It is not available on StreamNative Cloud. Please use it with caution.

Instead of using AoP, you are recommended to use RabbitMQ sink and source connectors to migrate data from RabbitMQ to Pulsar.

RabbitMQ is a popular, open-source messaging system that has been widely adopted for asynchronous service-to-service communication using a publisher-subscribe model.

RabbitMQ was built for single machines, but organizations today are dealing with more data than ever before. The amount of data created and consumed will continue to grow at a faster pace, with no signs of slowing down. Many organizations that have developed critical business applications on older messaging technologies are now facing scalability issues.

As a result, organizations are looking to cloud-native alternatives, and Apache Pulsar is increasingly the messaging technology of choice. Iterable, a marketing platform that sends large numbers of messages daily, made the jump from RabbitMQ when faced with flow control issues at high loads. Zhaopin, a popular online recruiting and career platform in China, experienced challenges with managing multiple messaging technologies using RabbitMQ. These are just two examples of organizations that have adopted Pulsar to modernize their messaging infrastructure and increase the resiliency and reliability of data at scale.

In this blog, we’ll discuss why organizations are choosing Apache Pulsar for messaging and how to leverage AMQP-on-Pulsar (AoP), a protocol handler for RabbitMQ, to enable an easy migration from RabbitMQ to Pulsar.

How Pulsar helps companies scale in the cloud

Apache Pulsar is cloud-native, enabling organizations to build scalable and reliable messaging and streaming applications in elastic cloud environments. First released to the open source community in 2016, Pulsar graduated as a top-level Apache Software Foundation project in 2018, and its adoption has skyrocketed since then. In 2021, Apache Pulsar was ranked as a top 5 ASF project.

While many think of Pulsar as a real-time data streaming solution, it was originally designed as a global messaging platform for Yahoo and can solve for use cases across both data streaming and messaging.

Pulsar is being increasingly adopted by organizations looking to solve scalability and reliability issues for asynchronous messaging and complex message queues. Let’s take a look at the key attributes driving Pulsar adoption:

1. Simplified operations with multi-tenancy - Pulsar’s built-in multi-tenant architecture enables organizations to securely deploy applications in a shared environment.
2. Elastic scalability - Pulsar’s decoupled storage and compute provides the ability to add new brokers and bookies independently, enabling seamless scalability.
3. Resiliency with geo-replication - Pulsar supports both asynchronous and synchronous geo-replication strategies across multiple data centers out of the box.
4. Cost-effective data retention - Pulsar’s tiered storage feature allows historical data to be offloaded to cloud-native storage and retain event streams for an indefinite period of time.

Next, we’ll look at a tool helping organizations make the move from RabbitMQ to Pulsar.

Introducing AoP: A turnkey protocol handler for RabbitMQ

Replacing any component of your software stack can be difficult, especially when it requires the migration of one or more applications that are integral to your business. For organizations looking to migrate from RabbitMQ to Pulsar, AMQP-on-Pulsar (AoP) is a protocol handler that helps make the transition easier.

AoP enables existing applications to communicate directly with Apache Pulsar using the same RabbitMQ client library: no API changes, and no changes to your existing code base. Just add the AoP protocol handler to your existing Pulsar cluster and you can run your existing application code “as is” with a few minor configuration changes. This will allow you to easily leverage Pulsar’s powerful features, such as tiered storage and infinite event stream retention.

How to Get Started with AoP

Getting started with AoP is a simple process. Follow the step-by-step instructions below to configure Apache Pulsar to support RabbitMQ:

1. Download and unzip the latest binary version of Apache Pulsar from the downloads page. We will refer to this unzipped folder as $PULSAR_HOME. As of this writing, the most recent stable version is 2.9.1.
2. Download the latest binary version of the AoP connector from the releases page, and copy it into $PULSAR_HOME/protocols directory. As of this writing, the most recent stable version is 2
3. .9.1.2.
4. Configure the Pulsar broker to run the AoP protocol handler as a plugin. Add the following configs to the Pulsar broker’s configuration file, e.g., broker.conf or standalone.conf if you are planning on running the broker in standalone mode (this is most likely the case in a laptop/developer environment).

4. Add messagingProtocols and protocolHandlerDirectory properties to the Pulsar Broker configuration file. For AoP, the value for messagingProtocols is amqp; the value for protocolHandlerDirectory is the directory where you downloaded the AoP NAR file.
5. Set AMQP service listeners in the Pulsar broker configuration file. Note that the hostname value in listeners should be the same as Pulsar broker's advertisedAddress, e.g., `` amqpListeners=amqp://127.0.0.1:5672 advertisedAddress=127.0.0.1
6. Start your Pulsar broker with the above configuration. For more details, refer to the AoP guide. If you made all of your changes to the standalone.conf file, then you will need to start Pulsar in standalone mode using the following command: “$PULSAR_HOME/bin/pulsar standalone
7. Create a namespace for the AMQP vhost using the following command: $PULSAR_HOME/bin/pulsar-admin namespaces create -b 1 public/vhost
8. Increase the data retention policy for the namespace you just created to 100 MB worth of data or 2 days using the following command: $PULSAR_HOME/bin/pulsar-admin namespaces set-retention -s 100M -t 2d public/vhost
9. Now you are ready to go and can test the AoP protocol handler using a RabbitMQ client (version 5.8.0 or newer is recommended) inside a Java program.
10. Once steps 1-9 have been completed, change the broker URL used by the RabbitMQ clients inside your application to the address you specified in the amqpListeners property, e.g., amqp://:5672. That will enable your application to work with Apache Pulsar instead of RabbitMQ.

Congratulations! After completing the steps above, you’ve configured Pulsar to support RabbitMQ. Now we’ll discuss one approach for migrating a production application from RabbitMQ to AoP.

Best Practices for Migrating from RabbitMQ to AoP

While there are several options to transition from RabbitMQ to Pulsar, in this example we’ll walk you through a migration process that includes establishing a parallel testing environment. A parallel testing environment helps mitigate potential migration risks for mission-critical applications.

Let’s take a closer look:

1. Stand-up a new Apache Pulsar cluster using one of the documented installation methods. (We recommend deploying to a Kubernetes environment using the Helm charts provided with the open-source distribution.)
2. Enable AMQP support for Pulsar. (You can find a step-by-step for this here.)
3. Replicate the topic structure for a single application on the Pulsar cluster so you have a place to publish the application data.
4. Create a new branch of the application code, and change its configuration to end messages to Pulsar instead of RabbitMQ.
5. Run the re-configured application instance along with an instance of the application configured to write to a RabbitMQ test environment.
6. Compare the contents of the topics.

This migration strategy allows you to gain a high degree of confidence that your existing application will run as expected before going live with the migration. Furthermore, your existing RabbitMQ environment can serve as a stand-by environment in the event of any unforeseen issues, allowing you to revert back to a safe environment if necessary.

Once you are confident with the AMQP setup on Pulsar, we recommend you begin the migration process on an application-by-application basis. This also allows you to identify any potential issues on a smaller scale.

Additional Resources on AoP

The RabbitMQ connectors (and several other connectors) are fully open source and supported by the Apache Pulsar community. If you are interested in testing AMQP on Pulsar, you can check out open source Pulsar here or use StreamNative Cloud to get a Pulsar cluster running in minutes.

To learn more about AoP, please check out the following blogs:

• Announcing AMQP 1.0 Connector for Apache Pulsar - Read about how the connector enables seamless integration between Pulsar and AMQP.
• Announcing AMQP-on-Pulsar: Bring Native AMQP Protocol Support to Apache Pulsar - Learn more about AoP architecture and concepts.
• How Apache Pulsar Is Helping Iterable Scale Its Customer Engagement Platform - Read about why Iterable migrated from RabbitMQ to Pulsar and how they are building a new messaging platform on Pulsar.