Queues Are Just Subscriptions: Demystifying Shared and Failover Modes (Pulsar Guide for RabbitMQ/JMS Engineers 4/10)

‍TL;DR:

In Pulsar, you don’t create a “queue” – you create a subscription. By having multiple consumers share the same subscription, Pulsar will distribute messages among them (just like multiple consumers on a RabbitMQ queue). This is Pulsar’s Shared subscription mode, which provides load-balanced consumption. For high availability (active-passive consumers), Pulsar offers Failover subscriptions, where one consumer is active and others stand by to take over on failure. This post explains how these subscription modes work, how they correspond to the traditional queue semantics, and when to use each. After reading, you’ll understand that to Pulsar, a queue is essentially a named subscription with possibly many consumers attached, and how Pulsar manages who gets what message in different scenarios.

Revisiting Pulsar Subscriptions vs Queues

We established earlier that Pulsar uses the concept of a subscription to handle what we think of as queues. A subscription represents a group of consumers with a given name on a topic. If one consumer is attached to that subscription, it will get all messages (and ordering is preserved). If multiple consumers attach to the same subscription, Pulsar must decide how to split messages between them. Pulsar offers a few policies (subscription types) to govern this distribution. The two most relevant for “queuing” are Shared and Failover (there is also Exclusive, which is the default single-consumer case, and Key_Shared, which we will cover in a later post about ordering).

• Exclusive subscription: Only one consumer can attach at a time. If another tries, it gets an error. This is essentially a 1-to-1 mapping (like a JMS Queue that only one consumer can have at a time, or a JMS durable topic subscription being consumed by only one process). Exclusive is Pulsar’s default type; it ensures strict ordering and simplest semantics (no concurrency).
• Shared subscription: Multiple consumers can attach; Pulsar will round-robin or load-balance messages across them. This is the true “competing consumers” setup akin to multiple consumers on a RabbitMQ queue or multiple listeners on the same JMS Queue.
• Failover subscription: Multiple consumers can attach, but one is designated as the primary (active consumer) and receives all messages. Others sit idle (or if the topic is partitioned, each partition might have a primary on different consumers). If the primary dies or disconnects, one of the backups takes over and continues from where it left off. This provides high availability without duplicate processing.
• Key_Shared subscription: A variant of shared where messages with the same key always go to the same consumer, to preserve ordering per key. This one combines aspects of parallelism and ordering and will be discussed in the context of ordering in one of our future blog posts.

For this discussion, focus on Shared vs Failover, as they essentially cover the two major ways you might use a queue in a system:

• Shared: for scaling out throughput (many workers share the load of a queue).
• Failover: for hot standby (only one worker at a time, but seamlessly switch if it fails).

Shared Subscription: Pulsar’s Competing Consumers

When you create a subscription and attach multiple consumers to it in Shared mode (also called “round-robin” mode in some Pulsar documentation), the broker will deliver each message from the topic to only one of the consumers on that subscription, distributing messages in a round-robin or weighted round-robin manner. Essentially, the subscription behaves like a classic queue – each message goes to one consumer – and multiple consumers means parallel processing of different messages.

Analogy: This is just like having a single RabbitMQ queue with multiple consumers. RabbitMQ will deliver each message in the queue to one consumer, fairly balancing by prefetch, etc. Pulsar does the same for a shared sub: each message in the subscription backlog is given to one of the available consumers.

Some points about Shared subscription behavior:

• If a consumer hangs (doesn’t ack a message), that message will eventually be re-dispatched to another consumer (via ack timeout or if the consumer disconnects). So work isn’t lost; another consumer can pick it up, similar to how RabbitMQ requeues unacked messages if a consumer dies.
• If a new consumer joins a shared subscription, the broker will start including it in the distribution of new messages. If a consumer leaves, the broker redistributes any unacked messages that were on that consumer to others.
• Order is not guaranteed across different consumers in a shared subscription. If you care about ordering, you either stick to one consumer (Exclusive or Failover) or use Key_Shared (ensures ordering per key). Shared basically is aimed at throughput scaling at the cost of ordering.
• Shared subscriptions support parallel processing: Each consumer can process messages independently. If one is slow, others still get new messages. This can maximize throughput.

Use cases for Shared:

• Work queues: e.g., tasks that can be processed in parallel (transcoding jobs, sending emails, etc.). You create one subscription (like “task-queue”), spin up N consumer instances with that subscription name, and Pulsar will divide tasks among them – voila, a distributed work queue.
  
• Scaling message consumption: If one consumer can’t keep up with the topic’s message rate, add more consumers on the same subscription to increase aggregate throughput.

How to create a Shared subscription:
When using the Pulsar client API, you specify subscription type Shared. For instance:

If you omit subscriptionType, it defaults to Exclusive (only one consumer at a time). So explicitly set it to Shared if you plan to attach multiple consumers. All consumers should use the same subscription name and same topic obviously.

Under the hood, what’s happening is:

• The first consumer to subscribe with that name will create the subscription on the broker.
• Additional consumers join that existing subscription. The broker keeps track of all active consumers for the sub.
• For each message, the broker chooses a consumer (basically cycling through them) and sends the message to that consumer.
• The subscription’s backlog is decreased when a message is acknowledged by a consumer.
• If a consumer disconnects with messages unacked, those messages will be re-dispatched to remaining consumers.

From a RabbitMQ perspective, you can consider Pulsar’s topic as analogous to an exchange+queue that all consumers draw from. The difference: you didn’t have to explicitly declare a queue and bind – you simply used a subscription name. Pulsar took care of tracking the offsets for that subscription.

JMS perspective: JMS 2.0 introduced the idea of a Shared Durable Subscription for topics, which allowed multiple consumers on the same durable subscription (to load balance topic messages). That’s quite analogous to Pulsar’s shared subscription on a topic. For JMS Queues, multiple consumers inherently share the queue. So Pulsar’s Shared subscription is fulfilling the role of both these concepts: a group of consumers sharing the work of one message stream.

Failover Subscription: High Availability Consumer

In a Failover subscription, multiple consumers can attach, but Pulsar will only deliver messages to one “primary” consumer at a time. If that consumer disconnects or times out, Pulsar automatically fails over to the next consumer in line, which will then start receiving messages from where the previous one left off.

Think of Failover as an active-standby cluster. One consumer is doing all the work until it can’t, then a standby takes over. This is useful when you have a service where only one instance should be active (maybe because processing must be single-threaded or use a resource exclusively), but you want a hot backup to take over instantly on failure. It’s also useful to implement ordered processing with high availability – you want only one consumer at a time to preserve ordering, but still want fault tolerance.

How Pulsar picks the primary: When consumers connect in Failover mode, they have a priority (you can set a priority level, default often 0, and an internal lexicographical order on consumer names as a tiebreaker). The broker will choose the highest priority consumer as primary. If equal priority, the one that connected first (or lexicographically smallest name, depending on version) becomes primary. Others are essentially parked.

For a non-partitioned topic, it’s straightforward: Primary consumer gets 100% of messages. If it dies, next in line gets all new messages (and any that were unacked by the first consumer will be delivered to the new primary). For partitioned topics, each partition has its own primary assignment; the broker might distribute partitions among consumers if you have multiple partitions, but within each partition only one consumer is active. (This means in a failover subscription with a partitioned topic, you could actually have all consumers active – but each on different partitions. It’s more complex, but effectively it’s like each partition is a mini-topic with failover selection, possibly balancing partition primaries across consumers, as described in the docs.)

Behavior on failover: Let’s say Consumer A is primary, Consumer B is standby. A is chugging along. Suddenly A’s process crashes or network breaks. Pulsar notices A’s connection is gone; it then promotes B to primary. B will start receiving any messages that were next. If A had some messages in flight (unacked) when it died, those will become available to B (after what is effectively an immediate redelivery – Pulsar doesn’t wait for ack timeouts on failover; once A is gone, its unacked messages are free to deliver to B). This ensures minimal interruption – B can continue processing the queue almost where A left off.

Use cases for Failover:

• Situations where you want only one consumer to handle messages, perhaps because processing should not be parallel (maybe a legacy system can’t handle concurrent processing, or order must be absolutely preserved end-to-end).
• High availability: e.g., a singleton service that you run two instances of for redundancy, but only one should actually do work at a time. If the active one fails, the backup seamlessly takes over.
• Think of an example: processing bank transactions from a topic. You might decide to use one consumer to ensure they are strictly sequential (no parallel processing that could reorder things), but you want a standby instance in case the main one goes down, so you’re not stuck waiting for manual intervention. Failover subscription is perfect here.

Comparison to RabbitMQ: RabbitMQ doesn’t have an explicit “failover consumer” concept. If you wanted active-passive, you might just run one consumer at a time. If it dies, something else would have to start consuming. With HA queues (mirrored queues in RabbitMQ), multiple nodes have the data, but only one node’s consumers actually consume at a time. Achieving seamless failover of consumption is typically done at application level for Rabbit (like using heartbeats to detect a dead consumer and then starting another consumer). Pulsar builds that logic in – you can start two consumers and know one will be idle until needed.

Comparison to JMS: JMS does not have “failover subscriptions” per se, but many JMS brokers would effectively behave similarly if you have multiple consumers on a queue – one might get all messages if it has a higher priority or some brokers allow exclusive consumer concept. For example, ActiveMQ has an exclusive consumer feature for queues: one consumer gets all messages until it dies, then another takes over. Pulsar’s failover is akin to that but at the subscription level.

How to use Failover:

And similarly on another instance with consumerName “Consumer-B”. (Consumer names aren’t mandatory but help in logging and also tie-breakers for ordering sometimes.)

If you want to designate priority, there’s subscriptionInitialPosition (to set where to start) and more relevantly ConsumerBuilder.priorityLevel(int) to give one consumer a higher priority. Higher priority consumers will always take precedence in failover. If you set one consumer with priority 1 and another with 2, the one with 2 will always be chosen if connected, the other is basically ignored until priority 2 is gone. By default, all have priority 0 (so then it falls back to whoever connects first).

To test failover, you can simulate killing the primary consumer – you should see the second consumer’s receive() calls now return messages.

One thing to mention: in failover mode, only one consumer receives messages at a time, so you’re not scaling throughput here, just providing redundancy. If you attach two consumers because you thought it would speed things up – it won’t, because only one is active. For throughput scaling, use Shared.

Under the Hood: Cursor and Partition details

Each subscription (regardless of type) has a cursor – essentially a pointer in the partition log that tracks how far along consumption has gone. In a shared subscription, the cursor moves as messages are acknowledged (which can happen out of order if multiple consumers acknowledge at different times; the cursor might actually mark the lowest acked point plus bitmaps of acked/unacked above that – but that’s an internal detail). In an exclusive or failover subscription, since only one consumer is doing sequential work, the cursor just moves sequentially with acks (or cumulatively).

For failover, when primary switches to secondary, the same subscription cursor is now being consumed by the new consumer. It picks up wherever the cursor last was. Any messages that were delivered to the first consumer but not acked are still marked unacked in the subscription, so the broker knows to redeliver them to the new consumer.

Partitioned topics and failover:
If the topic has multiple partitions (say 4 partitions), and you have two consumers in failover, the broker could assign half the partitions to one consumer as primary for those, and the other half to second consumer as primary for those, in order to utilize both consumers (this is optional and based on how priorities or names are sorted). This way, even in failover mode, both consumers might actually be active – but each on different partitions – so you get some throughput scaling too. If one consumer fails, the other takes over all partitions. This is a neat Pulsar nuance: failover subs can give you a mix of HA and load-spreading across partitions. However, if you want strict ordering across the whole topic, you wouldn’t partition the topic in the first place.

Summing Up Shared vs Failover vs Exclusive

It’s helpful to summarize with an analogy:

• Exclusive: One cashier at a store, one line of customers. If that cashier is out, the store is closed until a new one arrives.
  
• Failover: Two cashiers are present, but only one’s counter is open and taking customers; the other is in the back room on standby. If the first one has to step away, the second immediately opens their counter and continues serving the line. Customers always see exactly one open counter, so they go in order to that one.
  
• Shared: Two (or more) cashiers actively open, each handling their own line (or a shared line that dispatches customers to them). Customers (messages) get assigned to whichever cashier is free next (round-robin). This way, customers are served faster in parallel, but they are not in one single ordered line – effectively each cashier has their portion of the load.
  

From RabbitMQ perspective:

• Exclusive = similar to having a single consumer on a queue.
  
• Failover = no direct analog managed by Rabbit, but you could simulate by ensuring only one consumer connects (and use client-side logic to failover).
  
• Shared = typical multiple consumers on a queue scenario.
  
  

Setting it in Pulsar and Best Practices

If you’re configuring via pulsar-client CLI for a quick test:

• There’s a flag for subscription type (-t Exclusive|Shared|Failover|Key_Shared). e.g., pulsar-client consume my-topic -s subName -t Shared -n 0 will allow multiple instances of that command to share messages.

A note on message ordering and Shared subscriptions: When using Shared, since ordering isn’t guaranteed, be mindful if message order matters to your application’s logic. If it does, you either need to include ordering info in the message (like a sequence number and have the consumer sort or detect out-of-order), or avoid parallel consumption for those streams, or use Key_Shared to at least order per key. Many use-cases (like processing independent tasks) don’t need global ordering, so Shared is fine.

A note on failover with multiple partitions: If you truly want a single consumer to handle all messages in order, don’t partition the topic (a single topic is single-partition by default). If you have a partitioned topic and want to use failover, be aware multiple consumers might each handle different partitions simultaneously. If that’s not desired, stick to 1 partition.

Key Takeaways

• “Queues” in Pulsar are achieved by shared subscriptions: To have a queue with multiple workers, simply create a subscription and start multiple consumers with that subscription name and SubscriptionType.Shared. Pulsar will load-balance the messages across them. You don’t create a separate queue object – the act of consumers sharing the subscription is what creates the queue-like behavior.
  
• Failover subscriptions provide exclusive consumption with automatic failover: Only one consumer receives messages until it fails, then the next takes over. Use this for scenarios requiring a single consumer processing stream with high availability.
  
• Exclusive vs Failover vs Shared: Exclusive (the default) ensures only one consumer – any additional consumer with the same subscription name is rejected. Failover allows standby consumers. Shared allows concurrent consumers. Choose based on your needs: concurrency vs ordering vs HA.
  
• The subscription name is the queue name: If you connect 5 consumers to topic “alpha” with subscription “orders-sub” in Shared mode, those 5 effectively form the “orders-sub” queue group for topic alpha. If you had another subscription “billing-sub” on the same topic, that’s a separate queue group (receiving its own copy of messages, like a separate RabbitMQ queue bound to the same exchange).
  
• JMS and Rabbit equivalence: Pulsar Shared subs = JMS queue with multiple consumers, or JMS shared durable subscription; Pulsar Failover subs = JMS exclusive consumer or concept of exclusive queue consumption with automatic handover (not natively in JMS, but ActiveMQ’s exclusive consumer feature is similar). From Rabbit’s view, a Pulsar shared subscription is just like how Rabbit distributes messages to multiple consumers on one queue.
  
• No manual ack requeue hassle: In RabbitMQ, if a consumer didn’t ack, you either had to rely on the death of consumer for requeue or use basic.nack. In Pulsar’s shared subscription, if a consumer disconnects or negative-acks, Pulsar will readily redeliver unacknowledged messages to another consumer. So the queue processing will continue. This is managed by Pulsar’s subscription state.
  
• Queues are durable via subscriptions: Because Pulsar subscriptions retain messages until acked, a shared subscription with zero consumers still holds messages (like a durable queue would) and any new consumer that comes in will get them. That’s similar to a RabbitMQ durable queue sitting around until a consumer attaches. Pulsar does that automatically – the subscription (queue) exists as long as it has messages or a consumer attached. (You can administratively delete a subscription if needed, analogous to deleting a queue.)

To wrap up, Pulsar’s subscription model might have seemed abstract at first, but now we see it maps cleanly to queue semantics. “Shared” is what gives Pulsar the power to act like a traditional queue system for distributing tasks. Meanwhile, “Failover” and “Exclusive” cover scenarios requiring strict ordering or single-consumer behavior.

In the next post, we’ll talk about message durability, retention, expiration, and dead-letter topics. In other words, what happens when messages aren’t consumed, how to not lose them, and how Pulsar handles things like TTL and DLQs compared to RabbitMQ’s similar features. If you’ve ever set a queue to expire messages or configured a Dead Letter Exchange in RabbitMQ, the Pulsar way of achieving that is coming up next!