Pulsar 101 for Queue Users: Queues, Topics, and Subscriptions Explained (Pulsar Guide for RabbitMQ/JMS Engineers 1/10)

TL;DR:

Apache Pulsar doesn’t use “queues” in the same way as RabbitMQ or JMS. Instead, Pulsar has topics that serve as the central pipeline for messages, and subscriptions that track consumption progress (acting like logical queues). In this post, we explain how a Pulsar topic plus a subscription can replicate the behavior of a queue or a JMS topic. By the end, you’ll understand Pulsar’s publish/subscribe model, how it retains messages, and why you don’t explicitly create queue objects in Pulsar. We’ll also run through a quick example with Pulsar’s CLI and client API to demonstrate these concepts in action.

Topics vs Queues: The Key Concept Shift

If you’re coming from RabbitMQ or JMS, you’re used to queues as named containers that hold messages until consumers retrieve them. In Pulsar, there is no separate queue object – the Pulsar equivalent is achieved via topics and subscriptions. All messages in Pulsar are published to a topic, and consumers receive messages by attaching to a subscription on that topic. A subscription is Pulsar’s mechanism for tracking which messages have been delivered and acknowledged for a group of consumers.

• Topic: In Pulsar, a topic is a category or feed name to which producers publish messages. It’s similar to a RabbitMQ exchange or JMS destination in that producers write to it. Topics in Pulsar can be persistent (durable storage) or non-persistent (in-memory), but by default we deal with persistent topics that durably store messages.
  
• Subscription: A subscription is like a named pointer into a topic’s message stream. Consumers specify a subscription name when they subscribe. Pulsar will then deliver messages on that topic to the consumer and mark them as delivered for that subscription. If no consumer is currently active, the subscription retains all messages until they can be delivered. This is analogous to a durable queue holding messages for a consumer group. In fact, from a queue user’s perspective, a subscription is the queue – it accumulates unacknowledged messages for later processing, providing load balancing or broadcast behavior depending on how many consumers attach to it.
  

A single Pulsar topic can have multiple subscriptions. Each subscription acts as an independent feed of the topic. If you create two subscriptions on the same topic (say “SubA” and “SubB”), each will receive all messages published to the topic (each subscription has its own backlog). This is how Pulsar implements a pub/sub pattern (multiple subscribers each get their own copy of each message). On the other hand, if multiple consumers share the same subscription name (e.g. two consumer processes both subscribing with name “SubA”), then they will form a consumer group and Pulsar will distribute messages of that subscription’s backlog among them – effectively acting like a queue with competing consumers. We’ll dive deeper into those modes in a later post, but it’s important to grasp that in Pulsar, “queues” don’t exist as standalone objects – they emerge from how you use subscription names.

JMS perspective: In JMS, you have the concepts of Queue (point-to-point) and Topic (publish-subscribe). Pulsar’s model unifies these. A Pulsar topic can do both – if you have one subscription (like one queue) all consumers can share it and get load-balanced messages (point-to-point), or if you create multiple independent subscriptions, each behaves like a JMS durable topic subscription receiving a copy of each message (pub-sub). You don’t decide upfront whether a Pulsar topic is “queue-like” or “topic-like” – it can handle both patterns simultaneously. This flexibility is initially confusing but powerful once understood.

How Pulsar Stores and Delivers Messages

One major difference from RabbitMQ: Pulsar topics are backed by a persistent log (by default). When a message is published to a Pulsar topic, the broker writes it to durable storage (Apache BookKeeper bookies) and it remains there until it’s acknowledged by all subscriptions that are consuming that topic. This means if you have a subscription with no active consumers, the messages will sit in storage (in a backlog) indefinitely until a consumer comes along to consume/acknowledge them. Pulsar will not drop messages for an inactive subscription unless you configure explicit expiration (TTL) – so it behaves like a durable queue that keeps data safe even if consumers are offline, similar to JMS durable subscriptions.

By contrast, in RabbitMQ, once a message reaches a queue, if no consumer is attached the message will still sit in RAM/disk – so that part is similar. But RabbitMQ will drop messages on a non-durable queue if the broker restarts or if TTLs expire, etc., whereas Pulsar’s storage is persistent by default (we’ll cover durability in a later post). The key takeaway is that Pulsar decouples the storage of messages from the delivery. The topic is the stored log of messages, and the subscription is a position in that log. As consumers acknowledge messages, the subscription’s position moves forward and Pulsar knows it can safely remove acknowledged data (or retain it if you’ve enabled a retention policy for replay).

Acknowledgments: Pulsar uses acknowledgments to know when it can remove messages from the subscription backlog. When a consumer has processed a message, it sends an acknowledgement to the broker. Pulsar then marks that message as delivered for that subscription, and once all subscriptions have acknowledged a given message, it can be deleted from storage (or archived if retention is enabled). If a message is never acknowledged (perhaps the consumer crashed), the message remains in the backlog and will be redelivered to the next consumer that comes along on that subscription (ensuring at-least-once delivery, which we’ll discuss in Post 3).

One nice aspect: you don’t have to explicitly “create” a subscription ahead of time in code. If a consumer subscribes to a topic with a new subscription name, Pulsar will automatically start a subscription with that name. The first time you run a consumer with subscriptionName = "my-subscription" on topic persistent://public/default/my-topic, Pulsar creates the subscription state and begins tracking message delivery for it. It’s analogous to how RabbitMQ’s queues might be declared, but in Pulsar it often happens implicitly on first use (unless you disabled auto-creation). Topics themselves can also auto-create when first referenced (depending on broker settings), meaning you might not have to run an admin command to make a topic in simple cases – publishing to or subscribing from my-topic can make it appear. (In production, you might pre-create topics or use namespace policies to control this behavior, but it’s a convenient feature for development.)

Quick Walkthrough: Producing and Consuming with a Subscription

Let’s solidify these ideas with a quick example. Suppose you want to use Pulsar to replicate a simple RabbitMQ work queue. In Rabbit, you’d have a queue (let’s call it “tasks”) and you’d basicPublish messages to that queue. In Pulsar, we’ll use a topic, say “tasks-topic”. We want one consumer group processing it (could be one or many consumers), so we’ll use one subscription name, say “tasks-subscription”.

Step 1: Produce messages to the topic. We can use Pulsar’s CLI or a client library to send messages. For example, using the Pulsar CLI producer:

# Using pulsar-client CLI to produce some messages to a topic

$ bin/pulsar-client produce persistent://public/default/tasks-topic -m "Task-1" -m "Task-2" -m "Task-3"

This will create the topic tasks-topic (if not already created) in the public/default namespace and publish three messages: “Task-1”, “Task-2”, “Task-3”. In Pulsar, topics have a fully qualified name including a tenant and namespace; public/default is the default namespace most beginners use.

Step 2: Consume messages with a subscription. Now let’s start a consumer to receive these tasks. We’ll give it the subscription name “tasks-subscription”:

$ bin/pulsar-client consume persistent://public/default/tasks-topic \

    -s "tasks-subscription" -n 0 -p Earliest -t Shared

Here, -s specifies the subscription name, -p Earliest sets the consumer to read from the beginning of the topic, and -n 0 means consume indefinitely. When this consumer starts, Pulsar will see that subscription “tasks-subscription” exists (if first time, it creates it) and begin delivering messages. Since we published three tasks, the consumer should receive Task-1, Task-2, Task-3. After processing each, it will ack (the CLI consumer acks messages after printing them). Pulsar then marks those messages as acknowledged on “tasks-subscription” and clears them from the backlog.

If we were to run another consumer with the same Shared or Key_Shared subscription (tasks-subscription) at the same time, Pulsar would distribute the messages between the two consumers. For instance, one consumer might get Task-1, the other gets Task-2, etc., in a round-robin fashion. That’s the queue/competing-consumer pattern. But if we instead run a second consumer with a different subscription name (say “tasks-subscription-2”), that second consumer will receive a full copy of all messages independently. In our example, starting a second subscription after the tasks were sent wouldn’t receive those three, but if we send more tasks, each subscription would get its own copy.

To illustrate, let’s do that:

# Start a second consumer with a different subscription

$ bin/pulsar-client consume persistent://public/default/tasks-topic \

    -s "tasks-subscription-2" -n 0 -t Shared

Now publish another message:

$ bin/pulsar-client produce persistent://public/default/tasks-topic -m "Task-4"

Now, both the first consumer (tasks-subscription) and the second consumer (tasks-subscription-2) will receive “Task-4”. They are on different subscriptions, so each maintains its own position in the topic. This mimics a pub-sub (fan-out) scenario: you effectively have two “queues” listening to the same topic. In RabbitMQ terms, it’s as if we had an exchange with two queues bound to it, so a message went to both queues. Pulsar did that internally with one topic and two subs.

Conversely, if we had launched multiple consumers all using the same subscription “tasks-subscription”, then only one of them would get each Task-4 message (preventing duplicate processing), similar to multiple workers pulling from one RabbitMQ queue.

When to Use Multiple Subscriptions vs Shared Consumers

You might be wondering: how do I choose to use separate subscriptions or not? It depends on your use case:

• Multiple independent subscriptions (each with its own name): Use this when you want to broadcast messages to multiple independent groups of consumers. For example, one microservice (with subscription “serviceA-sub”) might process every message for one purpose, and another microservice (“serviceB-sub”) might also need every message for a different purpose. Each subscription will get its own copy of each message. This is analogous to multiple RabbitMQ queues bound to a fanout or topic exchange – each queue (subscription) gets the message.
  
• Shared subscription (multiple consumers, same name): Use this when you have a pool of consumers that collectively handle the messages in a load-balanced way. This is the classic worker queue scenario: many consumers, but each message should be processed only once by one of them. In Pulsar, they simply share the same subscription name and set the subscription type to “Shared” (more on subscription types in Post 4) to compete for messages from that one subscription backlog. This is analogous to multiple consumers on one RabbitMQ queue – the queue distributes messages to one consumer each.
  

The beauty of Pulsar is that the topic is decoupled from these patterns. You don’t need separate physical queues for fan-out versus work distribution. It’s all about how you use subscription names. For someone used to JMS, think of Pulsar’s topic as either a JMS Topic or Queue depending on how you subscribe: if each consumer uses a different durable subscription name, it acts like a JMS Topic (each durable sub gets all messages). If consumers share a subscription, it acts like a JMS Queue (messages go to one consumer). In fact, the Pulsar <-> JMS mapping (via Starlight for JMS) literally treats a JMS Queue as a Pulsar topic with a single shared durable subscription under the hood.

Key Takeaways

• Topics are the core entity in Pulsar: producers write to topics, and topics store messages durably. There is no standalone “queue” object as in RabbitMQ; Pulsar topics + subscriptions cover that functionality.
  
• Subscriptions act like durable queues: a subscription retains unacknowledged messages and ensures consumers receive them. Each subscription has its own backlog of messages on a topic. A subscription can have one or many consumers attached.
  
• Multiple subscription names = pub-sub: If you create multiple subscriptions on the same topic, each behaves like an independent stream of all messages (fan-out). In other words, Pulsar can deliver one message to multiple subscriber groups without duplicating producers or topics.
  
• Shared subscription (one name, many consumers) = queue load-balancing: If consumers share the same subscription, Pulsar distributes messages among them (each message to only one consumer) – analogous to multiple consumers on one queue. You get parallel processing without double-handling of the same message.
  
• Durable by default: Pulsar’s persistent topics retain messages until acknowledged by the subscription. Consumers can be offline, and on return they will get the backlog. This is similar to JMS durable subscriptions and unlike non-durable transient queues – Pulsar defaults to durability so you don’t lose messages.
  
• No need to pre-create queues: You typically don’t pre-declare a “queue” in Pulsar. You decide on topic names and subscription names in your client, and Pulsar will manage the rest (topics and subs can auto-create on first use, unless locked down by config).
  

Armed with this knowledge, you’re ready to dive deeper. Next, we’ll explore how Pulsar handles the routing patterns that RabbitMQ implements with exchanges. If you’re wondering “how do I do a fanout or direct routing in Pulsar without exchanges?”, stay tuned for the next post!