What’s New in Apache Pulsar 2.6.4

What’s New in Apache Pulsar 2.6.4

We are excited to see the Apache Pulsar community has successfully released the 2.6.4 version! 10 contributors provided improvements and bug fixes that contributed to 16 PRs.

Highlights:

• Broker no longer delivers old messages after a topic is closed (#8634)
• AWS credentials are refreshed after expiry (#9387)
• Pulsar identifies when individual message deletes cause an unsynced cursor (#9732)

This blog walks through the most noteworthy changes. For the complete list including all enhancements and bug fixes, check out the Pulsar 2.6.4 Release Notes.

Notable enhancement

C++ client

C++ client supports multiple topic subscriptions across multiple namespaces (#9520)

Issue

Previously, you could not subscribe to different topics on different namespaces.

Resolution

• Move the check for namespace in MultiTopicsConsumerImpl to PatternMultiTopicsConsumerImpl that uses a regex subscription.
• Fix the existing tests for subscriptions on topics across different namespaces.

Notable bug fix

Broker

Pulsar guarantees security for clients using JWT (#9172)

Issue

Previously, it was possible for attackers to connect to Pulsar instances because the signature of the JWT was not validated when the token was set to none.

Resolution

Modified JWT to use parseClaimsJws instead of parse to get the token objects. Now, parseClaimsJws guarantees the correct security model for parsing signed JWTs.

Pulsar identifies when individual message deletes cause an unsynced cursor (#9732)

Issue

Previously, cursors were not being flushed when acknowledgements caused a dirty cursor. Instead of deleting the acknowledged messages, messages were redelivered.

Resolution

Fixed code to mark the individual acknowledgements and automatically trigger the flush of dirty cursors.

Pulsar can expire a range of messages (#9083)

Issue

Previously, only a single message expired after an expiry check. As a result, many expired messages remained in a subscription and were delivered to consumers after the expiry time.

Resolution

Modified OpFindNewest to jump to a valid position, which allows PersistentMessageExpiryMonitor to find the best range of messages to expire.

Pulsar allows manual (forced) topic deletion after removing non-durable subscriptions (#7356)

Issue

Previously, during the removal of non-durable subscriptions, there was a race condition that left a topic in a state where you could not delete it until it was unloaded or reloaded.

Resolution

Fixed the race condition by setting the topic fence before performing any delete operations and reverting the topic state after the delete operations.

Broker no longer delivers old messages after a topic is closed (#8634)

Issue

Previously, it was possible to re-deliver very old messages if a topic was not gracefully closed. The cursor rolled back to the last persisted position and triggered the re-delivery of those messages.

Resolution

Fixed the redelivery of messages by setting a time-bound period after which all cursor updates are flushed on the disk.

Batch index acknowledgement data is no longer persisted (#9504)

Issue

Previously, the batch index acknowledgement data persisted because batchDeletedIndexInfoBuilder generated the batch index acknowledgement data but did not clear the current set before adding the delete set.

Resolution

Fixed by clearing the delete set before adding a new delete set.

Closed ledger deletes after expiration (#9136)

Issue

Previously, a closed ledger (with no incoming traffic) could fail to delete after expiring because the read position of the cursor still points to the last entry of the closed ledger.

Resolution

Updated behavior when closing the current ledger. Now, when the cursor's mark-delete position points to the last entry of the current ledger, the read position is moved to the newly created ledger.

Tiered storage

AWS credentials are refreshed after expiry (#9387)

Issue

Previously, expired AWS credentials were reused. With the refactor of Azure support, a regression occurred where the AWS credentials are fetched once and then used through the entire process.

Resolution

The AWS credential provider chain takes care of the credential refresh. When integrating with JClouds, you still need to return a new set of credentials each time.

Java client

Compression applied during schema preparation (#9396)

Issue

Previously, compression was not applied during deferred schema preparation and the consumer could receive an uncompressed message and then fail.

Resolution

Fixed by enforcing compression during the schema preparation.

Get involved

To get started, you can download Pulsar directly or you can spin up for a Pulsar cluster on StreamNative Cloud with a free 30-day trial of StreamNative Cloud in which Pulsar 2.6.4 changes are shipped! Moreover, we offer technical consulting and expert training to help get your organization started. As always, we are highly responsive to your feedback. Feel free to contact us if you have any questions at any time. Look forward to hearing from you and stay tuned for the next Pulsar release!

‍