Pulsar Newbie Guide for Kafka Engineers (Part 4): Subscriptions & Consumers

‍TL;DR

This post dives into how Apache Pulsar handles subscriptions and consumers, which is Pulsar’s equivalent to Kafka’s consumer groups. Pulsar requires consumers to specify a subscription name, which acts like a consumer group ID in Kafka. You can have multiple subscriptions on the same topic (for multi-group fan-out) and each subscription can have one or more consumer instances attached. Pulsar offers four subscription types – Exclusive, Failover, Shared, and Key_Shared – that determine how messages are delivered to consumers. Exclusive (the default) and Failover ensure only one consumer (or one active consumer at a time) receives all messages (preserving order like Kafka) of one or multiple partitions. Shared and Key_Shared allow multiple consumers to split the messages of a partition: Shared distributes messages round-robin (like a queue, higher throughput but no global order guarantee), while Key_Shared also distributes messages but guarantees ordering per message key. Pulsar’s broker tracks a subscription cursor (like an offset) for each subscription to maintain where consumers left off, and unacknowledged messages form a backlog (analogous to Kafka’s consumer lag) that you can monitor. In short, Pulsar’s flexible subscription model lets you achieve Kafka-like streaming and RabbitMQ-like queuing patterns on the same platform.

Understanding Pulsar Subscriptions vs Kafka Consumer Groups

If you come from Kafka, you’re used to consumer groups: a named group of consumers where each Kafka partition is consumed by one member of the group. Pulsar approaches this concept with subscriptions. A subscription in Pulsar is essentially a named rule for consuming a topic – think of it as a durable consumer group on a single topic. Consumers subscribe by specifying a subscription name, and Pulsar will ensure messages are delivered according to the subscription’s type (more on types soon). Under the hood, when a subscription is created, Pulsar sets up a cursor to track the subscription’s position in the topic. This cursor is stored durably (in BookKeeper) so that if the consumer disconnects or the broker restarts, the subscription’s last read position is remembered. In Kafka, the consumer group’s offsets serve a similar purpose (often stored in an internal topic). In Pulsar, the broker itself manages the offsets (cursors), which simplifies offset management – you don’t need an external store or to manually commit offsets, it’s handled by the act of acknowledging messages.

Because Pulsar decouples the subscription from the physical consumer, you can have multiple subscriptions on one topic just by using different subscription names. Each subscription name represents an independent feed of the topic. For example, if you have two separate services that need the same data, you can have one Pulsar topic with two subscriptions (say “serviceA” and “serviceB”), and each subscription will get every message published – effectively duplicating the stream, like two separate Kafka consumer groups reading the same topic. Pulsar keeps track of a cursor for each subscription, and each subscription has its own backlog (messages published to the topic that have not yet been acknowledged on that subscription). This is powerful: it means Pulsar inherently supports fan-out (pub-sub) as well as work-queue sharing patterns on the same data. You could have one subscription where only one consumer reads all messages (stream processing), and another subscription on the same topic where a pool of consumers share the messages (distributed queue processing).

Let’s clarify some terminology: acknowledgment in Pulsar is the act of a consumer confirming it has processed a message. When a message is acknowledged, the subscription’s cursor moves forward, and the message is considered consumed for that subscription. Acking in Pulsar is analogous to committing an offset in Kafka, but it’s automatic when you use Pulsar’s APIs (or CLI) unless you disable auto-ack. Importantly, Pulsar supports two acknowledgment modes: individual acks (ack each message) and cumulative acks (acknowledge all messages up to a given position in one go). Cumulative acks are useful in Exclusive/Failover subscriptions to advance the cursor in bulk, but they aren’t supported in Shared mode (since out-of-order consumption makes it tricky). In practice, individual ack is common and is what the Pulsar client does by default. Unacknowledged messages remain in the subscription backlog and will be redelivered later or to other consumers if possible. The backlog is essentially the number of messages pending acknowledgment – similar to the idea of “consumer lag” in Kafka (how many messages behind the tip of the log you are). You can view the backlog and other stats with pulsar-admin topics stats as we saw in the first blog, which shows the subscription cursor position and backlog for each subscription. Pulsar will retain messages as long as there is at least one subscription that hasn’t acknowledged them (or until retention limits kick in). If a topic has no subscriptions or if all subscriptions have acknowledged a message, that message can be deleted (Pulsar doesn’t require a fixed retention period if messages are consumed, unless you’ve enabled time or size-based retention). This is a key difference: Kafka stores messages for a time window regardless of consumption, whereas Pulsar by default can delete acknowledged data (making it more storage-efficient for queue use cases), while still allowing you to configure retention to keep data for replay if needed.

Another difference is how new consumers start consuming. In Kafka, when a new consumer group is created, it has a auto.offset.reset policy (earliest or latest). In Pulsar, when you subscribe to a topic with a new subscription name, you also choose where to start: by default it’s at the latest position (meaning you only get messages published from that point onwards), but you can specify -p Earliest (or subscriptionInitialPosition in the client API) to consume from the beginning of the topic’s backlog. We demonstrated this in our first blog post by using -p Earliest for the console consumer to read existing messages. So remember, if you create a new subscription and want to replay from the start, specify the initial position accordingly; otherwise you might think “nothing is coming through” simply because by default it’s tailing new messages only.

In summary, think of a Pulsar subscription as the combination of Kafka’s consumer group concept and offset tracking mechanism, managed for you by Pulsar’s broker. Next, let’s explore the different subscription types that Pulsar offers – this is where Pulsar really shines in flexibility compared to Kafka.

Subscription Types in Pulsar

Pulsar has four subscription types: Exclusive, Failover, Shared, and Key_Shared. These define how messages are delivered when multiple consumers attach to the same subscription name on a topic. (If consumers use different subscription names, they’re completely isolated – each subscription sees all messages independently, as discussed.) By choosing a subscription type, you control whether a subscription behaves more like a traditional stream (one consumer getting all messages in order) or a queue (multiple consumers dividing up messages) or a blend of both. Let’s break down each type:

Exclusive

Exclusive is the default subscription type in Pulsar. As the name suggests, an Exclusive subscription only allows one consumer at a time to attach to all the partitions of a topic. If a second consumer tries to subscribe with the same subscription name while an active consumer is already attached, the broker will refuse the second consumer (the consumer will get an error indicating the subscription is already taken). This is akin to a Kafka consumer group with a single member (and Kafka would similarly not use a second member if there’s only one partition – it would just sit idle). Exclusive subscriptions guarantee that the entire topic’s messages go to one consumer, preserving the message order end-to-end, since no other consumer is concurrently receiving messages.

Because only one consumer can consume, Exclusive subscriptions aren’t about scaling out consumption; they are useful for strict ordering or when you truly only want one consumer processing a given stream of data. One common pattern is to use multiple exclusive subscriptions on the same topic to implement a pub-sub fan-out: for example, you might have two different services that need the data from topic X. You can have Service A use subscription “subA” (exclusive) and Service B use subscription “subB” (exclusive). Both services will get all messages from topic X independently (since they are on different subscriptions), each in order for themselves. This is exactly how Pulsar enables pub-sub – multiple exclusive subscriptions on the same topic – analogous to having multiple Kafka consumer groups reading the same topic. The difference is that in Pulsar, the broker tracks the cursor for each subscription and retains messages until each subscription acknowledges them, so you get durable pub-sub with one topic rather than duplicating data. In short, Exclusive = one consumer at a time, simplest model. It’s also the fallback: if you don’t specify a subscription type, you’ll get Exclusive by default.

Failover

Failover subscriptions allow multiple consumers to attach to the same subscription, but still only one consumer actively receives messages from a partition at any given time. The idea is to have a primary consumer and one or more backup consumers. If the primary (master) consumer disconnects or becomes unreachable, one of the backups is promoted to be the new primary and continues consuming from where the previous one left off. This provides high availability for consumption: if you have a critical processing pipeline, you can run a standby consumer that will take over automatically if the main consumer fails, minimizing downtime or data buildup.

How does Pulsar choose the primary and the failover order? By default, it’s based on the order in which consumers subscribe (or you can assign each consumer a priority level). The first consumer to attach becomes the master for the topic (or for each partition of the topic, if partitioned – more on that in a second). Second becomes the next in line, and so on. All consumers beyond the first are in a standby mode – they are connected and ready, but they do not receive messages while the master is active. They typically sit idle (Pulsar might send occasional heartbeats to them to know they’re alive, but no actual message traffic).

When the master consumer disconnects (or you deliberately close it, or it crashes), Pulsar will automatically start delivering messages to the next consumer in line. Any messages that were sent to the original consumer but not acknowledged will be redelivered to the new consumer as well, so no messages are lost. The newly promoted consumer continues from the last acknowledged position of the previous one, maintaining continuity. Message order is preserved under Failover because at any given time, each partition’s messages are processed by a single consumer. It’s similar to Exclusive in that sense (one-at-a-time consumption), except it permits a standby to take over instantly on failure. In fact, from an ordering standpoint, Exclusive and Failover are the same (strict ordering); the difference is Failover gives you redundancy.

A key detail for those coming from Kafka: with partitioned topics, Failover will assign the master role per partition. This means if you have a topic with 10 partitions and two consumers in a failover subscription, Pulsar will try to balance such that each consumer is master for some of the partitions. For example, consumer A might be primary for partitions 0-4 and consumer B for partitions 5-9 (the assignment is done by the broker, trying to even it out). In that case, both consumers are actually active simultaneously, but on different partitions. If one consumer dies, the other will take over all partitions. This behavior is analogous to Kafka’s consumer group rebalancing (each consumer gets some partitions). However, if the topic is non-partitioned (a single partition essentially), then only one consumer (the first or highest priority) gets all the messages, and the others truly get nothing until failover occurs. So, Failover mode can act both as a pure hot-standby (in single-partition topics) or as a load-sharing mechanism across partitions (in multi-partition topics). The main point remains: for each partition, one consumer is doing the work at a time. This guarantees order per partition and no duplicate processing. (If a new consumer with higher priority joins, it can even preempt and become the master for partitions, but that’s an edge scenario.)

In practice, you’d use Failover when you need reliability – e.g., you have a critical consumer and you want a backup to seamlessly continue if the primary fails. It’s common in scenarios where processing order matters but you also want quick failover for HA. If you tested this with the Pulsar CLI, you could do something like:

• Terminal 1: pulsar-client consume -s mySub -t Failover -p Earliest -n 0 persistent://public/default/my-topic
• Terminal 2: pulsar-client consume -s mySub -t Failover -p Earliest -n 0 persistent://public/default/my-topic
  

Both will connect. You then publish some messages (using pulsar-client produce). You’ll notice only one of the two terminals is printing the messages – that’s the master. If you stop Terminal 1 (the master), Terminal 2 will immediately start receiving any new messages. Any messages that Terminal 1 did not ack before it went down will be redelivered to Terminal 2 as well. This behavior confirms the failover: one active consumer at a time, automatic hand-off on failure. This is different from Kafka where if a consumer in a group dies, there is a rebalance delay and then other consumers resume partitions; Pulsar’s failover is near-instant for new messages because the standby is already connected and ready.

One caveat: if a failover happens at an awkward time, there is a possibility of a couple of messages being processed out of order or twice (for example, the old consumer got a batch of messages but crashed before acking some, and the new consumer might receive some of those messages again while the old one may have actually processed some before crashing). Pulsar’s documentation notes that in some cases you may see a duplicate or an out-of-order message around the switchover. But in general, failover mode is designed to hand off smoothly with minimal duplication.

Shared (Round-Robin)

With a Shared subscription, multiple consumers can connect to the same subscription on a topic partition and receive messages concurrently. Unlike Exclusive/Failover, where only one consumer gets all messages, in Shared mode the broker will round-robin dispatch messages to consumers. Each message from the topic goes to one of the consumers in the group (never to more than one), distributing the load. Effectively, this turns your topic + subscription into a work queue – multiple consumers are pulling from the same queue of messages, each handling different messages in parallel. This is great for scaling out processing: if one consumer instance isn’t fast enough to keep up with the topic’s throughput, you can add a second, third, etc., on the same subscription and Pulsar will spread the messages among them.

Because messages are distributed, ordering is not guaranteed across the subscription as a whole. If message A and then B are published, it’s possible A goes to Consumer 1 and B goes to Consumer 2, and Consumer 2 might process B before Consumer 1 processes A. There’s no coordination to preserve publish order in a Shared subscription – the goal is throughput and load balancing. If ordering is important, Shared might not be the right choice (or you’d need to ensure all related messages go to the same consumer, which is what Key_Shared is for). Shared subs also do not support cumulative ack (since each consumer may be at a different position in the stream, there’s no single “up to X” point that makes sense to ack collectively) – consumers should ack messages individually.

One of the big advantages of Shared mode is how it handles slow or stuck consumers. Since each message is delivered to one consumer at a time, if that consumer fails to acknowledge (maybe it died or is hanging), Pulsar can detect that (via ack timeouts or the TCP connection closing) and will redeliver those unacked messages to another consumer in the group. For example, if Consumer A received Message 5 but never acked it (maybe Consumer A crashed), after a timeout, Pulsar will requeue Message 5 and send it to Consumer B (assuming Consumer B is healthy). This ensures that a bad consumer doesn’t black-hole messages – the work will be picked up by someone else. Meanwhile, other messages that were sent to other consumers can continue being processed; one slow consumer doesn’t block the others. This contrasts with Kafka’s model where if a consumer in a group slows down on a partition, that partition’s consumption lags behind (since Kafka won’t hand those messages to a different consumer unless the first consumer is considered dead and a rebalance happens). Pulsar’s Shared mode provides a more dynamic load balancing: each message is assigned to a consumer, and if that consumer can’t handle it, it can be reassigned. This is why Pulsar can achieve true queue semantics on a stream. It’s very much like how a RabbitMQ work queue would behave – many consumers pulling tasks off a queue, each ACKing tasks as done, and the system requeuing unacked tasks if a worker goes away.

In terms of usage, Shared subscriptions are ideal when you have independent messages and you want to maximize parallel processing. If ordering doesn’t matter (or you only care about per-message handling, not sequence), use Shared to scale out. For example, imagine a thumbnail generation service where each message is “generate a thumbnail for image X”. The order doesn’t matter at all – you just want to process as many as possible in parallel. A Pulsar topic with a Shared subscription and many consumers allows you to spin up N workers and they’ll automatically load balance the tasks. Each consumer will acknowledge as it finishes a message; the subscription’s cursor advances per message as a result of those acks (the cursor essentially will mark the message as consumed when acked, but since messages might be out-of-order, the cursor might have holes – which is fine, those holes are the backlog of unacked messages). The Pulsar admin stats will show how many messages are in backlog (i.e., not yet acked). In a healthy steady state, backlog stays near zero as consumers keep up; if consumers fall behind, backlog grows (like a queue depth). You can always add more consumers to that subscription to catch up if needed – Pulsar will incorporate them and start sharing messages with the new consumers immediately.

Key_Shared

Key_Shared is the newest addition (relative to others) to Pulsar’s subscription types. It’s like an enhanced version of Shared that strikes a balance between ordering and parallelism. In a Key_Shared subscription, multiple consumers can attach and all will receive messages, but messages that share the same key will always go to the same consumer. In other words, Pulsar will hash or map message keys to specific consumers, and ensure that the order of messages for each key is preserved on that consumer. If that consumer disconnects, the messages for that key will be routed to another consumer, but always in a way that maintains the ordering from the last acked message onwards (Pulsar will not suddenly deliver older unacked messages of that key to a new consumer out of order).

This mode is extremely useful when your messages have some natural key (like user ID, or order ID, or device ID) and you want to ensure all messages for that entity are processed in order, but you don’t care about ordering across different entities. Kafka achieves something similar by requiring you to put all messages for an entity on the same partition – which then ties parallelism to partition count. Pulsar’s Key_Shared does this dynamically with consumers: you could have a single topic (single partition if you want) and still scale out consumption by key. The broker handles the assignment of keys to consumers. In fact, if you add more consumers, Pulsar can redistribute the hash ranges of keys among them automatically. If a consumer leaves, its key range is taken over by others. This all happens behind the scenes, giving you the effect of partitioning without manual partition management.

From the application perspective, Key_Shared means: “I have multiple consumers, but I want to ensure no two consumers ever process the same key’s messages concurrently or out of order.” It provides ordering per key and load-balancing across keys. A classic use case might be an event stream where events are tagged with a customer ID and you want per-customer ordering (maybe to avoid race conditions updating a customer’s state), but you also want to process different customers in parallel. With Kafka, you’d need as many partitions as you have parallelism (and all messages for a customer must go to the same partition). With Pulsar Key_Shared, you can spin up multiple consumers for a topic and Pulsar will ensure messages with the same key always go to the same consumer. For example, imagine tracking user activity where each message has a user_id as the key: With 10 consumers: All events for key:"user_789" will consistently go to the same consumer (let's say Consumer #3). Other users like key:"user_456" and key:"user_123" will each be consistently routed to their own assigned consumers. When you scale to 20 consumers: key:"user_789" might get reassigned to Consumer #7, but all their events will still go to just that one consumer. This gives you parallel processing across different users while maintaining strict ordering per individual user. The key-to-consumer assignment is handled automatically by Pulsar's hash-based distribution. This is handled by one of the available key distribution strategies (like auto-split ranges or consistent hashing), but you usually don’t need to worry about the exact algorithm as a user – just know that it balances keys.

In summary, Key_Shared = like Shared (multiple consumers in parallel), but with ordering guaranteed on a per-key basis. It’s the best of both worlds for many scenarios, giving you scaling with correctness. Key_Shared is often recommended when your use case can leverage message keys to delineate order boundaries – for instance, any stateful processing per entity should use Key_Shared if you want to scale out that processing. If ordering doesn’t matter at all, plain Shared is fine; if global order matters, you’d stick to Exclusive/Failover. Key_Shared fills the gap of “order matters per entity, but not globally.”

Putting it All Together

The beauty of Pulsar is that you can mix and match these subscription types to fit your needs, even on the same topic. For example, you could have one subscription on a topic using Key_Shared with 5 consumers processing events in parallel, and another subscription on the same topic using Exclusive to feed a separate system that needs the full ordered stream. The publisher only writes the message once, but Pulsar can deliver it in multiple ways to different subscribers. This is something not easily done in Kafka without duplicating data or using external systems – Pulsar’s design cleanly separates the publish side from the subscribe side through these named subscriptions.

To reinforce these concepts, it’s worth comparing with Kafka’s approach:

• In Kafka, if you want to do pub-sub (fan-out) you typically create multiple consumer groups. In Pulsar, you create multiple subscriptions (which is effectively the same idea). Each subscription has its own cursor and backlog.
  
• In Kafka, if you want a work-queue pattern, you’d create a consumer group with multiple consumers. Kafka will then assign partitions to consumers (can’t have more consumers than partitions effectively) and you get parallelism at the partition level, but strict ordering within each partition. If one message in a partition is slow or causes an error, it blocks everything behind it in that partition until it’s handled or skipped. In Pulsar, for work-queue, you use a Shared subscription on a topic (which could even be a single partition topic). You get parallelism per message, not just per partition, and a slow message doesn’t block others – it can be retried elsewhere while other messages still flow to other consumers. This is a major difference in the consumption model and is one of Pulsar’s key advantages for certain workloads.
  
• Key_Shared doesn’t really have a direct equivalent in Kafka. Kafka would require you partition by key to get key-ordering, but that then ties you to a static number of partitions and possibly uneven key distribution. Pulsar’s Key_Shared is more flexible and dynamic in that regard (you can increase consumers on the fly and it will redistribute keys, whereas Kafka partition count is fixed once topic is created, unless you manually add partitions which is a heavyweight operation and can cause ordering issues of its own for existing keys).

Conclusion

In this part, we corrected and clarified Pulsar’s subscription and consumer mechanics. We learned that Pulsar’s subscription name is analogous to Kafka’s consumer group – it’s how Pulsar tracks a consumer or group of consumers reading a topic. Pulsar’s broker maintains a subscription cursor for each subscription to know which messages have been acknowledged (processed), ensuring durability and allowing consumers to pick up where they left off after disconnects. We also reviewed the four subscription types in Pulsar and how they map to messaging patterns:

• Exclusive: Single-consumer, like Kafka only allows one consumer (ensures total order).
  
• Failover: Single active consumer with standby failovers, same as Kafka (ensures order, with quick failover on consumer loss).
  
• Shared: Multiple consumers, competing for messages in a queue-like fashion (higher throughput via parallelism, no overall order guarantee, built-in replay of unacked messages).
  
• Key_Shared: Multiple consumers with ordering per key (best for parallel processing when per-key order matters, effectively combining ordering and load balancing).

Pulsar gives you the freedom to use the right tool for the job – or even use both at the same time on the same data. You can have stream processing and queue processing co-exist on one topic through different subscriptions. This flexibility is one of the reasons Kafka engineers find Pulsar intriguing: it’s like having Kafka and RabbitMQ in one system. By leveraging subscription types, you can implement complex messaging workflows without deploying multiple platforms.

Now that you understand subscriptions and consumers in Pulsar, you’re well-equipped to build systems that take advantage of Pulsar’s dual nature of streaming and queuing. In the next part of the Pulsar Newbie Guide, we’ll continue our journey (stay tuned!). Meanwhile, feel free to experiment with subscription settings in a test environment to solidify your understanding – Pulsar’s CLI and admin tools make it easy to observe how messages flow under each mode. Happy Pulsar-ing!

‍

-------------------------------------------------------------------------------------------------------------------

‍

Want to go deeper into real-time data and streaming architectures? Join us at the Data Streaming Summit San Francisco 2025 on September 29–30 at the Grand Hyatt at SFO.

30+ sessions | 4 tracks | Real-world insights from OpenAI, Netflix, LinkedIn, Paypal, Uber, AWS, Google, Motorq, Databricks, Ververica, Confluent & more!

[Explore the Full Agenda]

[Register Now]

‍