StreamNative Tiered Storage v2 is a StreamNative-developed implementation for reading Pulsar data offloaded to object storage. The capabilities described here belong to StreamNative's implementation, not a new feature release of the Apache Pulsar community offloader.
For teams backfilling applications, replaying events, or consuming accumulated backlogs, it provides three practical benefits: higher aggregate consumption throughput, fewer redundant object-storage reads, and continued access to existing offloaded data without rewriting it in the default legacy-compatible format.
In a six-subscription catch-up test, average aggregate consumption throughput increased from 12.4k to 76.6k messages per second, approximately 6× the previous offload implementation. This compares two offload readers; it does not measure individual read latency or compare performance with BookKeeper reads.
Why concurrent cold-data reads need a different approach
Pulsar tiered storage makes longer retention economical by offloading data from BookKeeper to object storage. Here, cold data means offloaded data. When several subscriptions consume it from different positions, the previous reader can repeat downloads and scans. Synchronous reads also occupy workers assigned by ledger ID, so busy ledgers sharing one worker can queue while other workers have spare capacity.
StreamNative Tiered Storage v2 combines asynchronous reads, finer-grained indexing, shared caching, and read-ahead to improve aggregate consumption throughput and reduce redundant work. The comparison below shows how these changes fit together.
How StreamNative improves the offload read path
Figure 1. The offload read path before and after the StreamNative implementation changes. Arrows show a conceptual request path; cache hits avoid remote reads.
Overlap remote reads to support higher throughput
The asynchronous pipeline removes the old reader's fixed per-ledger worker assignment. Independent requests can overlap within resource limits, reducing blocking between ledgers. S3 and Azure use native asynchronous clients; the current GCS adapter wraps its synchronous SDK in a bounded pool.
Reuse fetched data across subscriptions
A broker-wide chunk cache lets subscriptions reuse downloaded data and share in-flight fetches. Sequential read-ahead places upcoming chunks in the same cache. This reduces duplicate requests and helps sustain consumption throughput; the benefit depends on access overlap and cache capacity.
Reduce scanning with finer-grained indexing
New offloads add a sparse index, allowing the reader to start closer to the requested entry. Existing offloads remain readable with their original block-level index and can benefit from the updated read path. The additional index is written with new data; it is not retroactively added to old objects.
Separate cache and in-flight read budgets control memory and remote-read pressure. Backpressure limits additional work when concurrent demand exceeds the available capacity.
Remove the retired Apache jclouds integration
Tiered Storage v2 replaces the retired Apache jclouds integration with cloud-provider SDKs, removing the legacy jclouds dependency stack and enabling ongoing dependency and security updates.
Existing data stays readable. New data gains finer-grained indexing.
Adoption does not require rewriting existing offloaded objects when using the default legacy-compatible storage format. The new reader supports both existing offloaded data and data written with the additional sparse index:
| Offloaded data | How the new reader accesses it | Benefit |
|---|---|---|
| Existing data with the original block-level index | Locates the block and scans to the requested entries through the updated read path | Shared caching and read-ahead can improve reads without rewriting the data |
| Newly offloaded data with the additional sparse index | Uses finer-grained offsets to start closer to the requested entries | The same read-path improvements, plus less scanning to locate data |
Backward compatibility preserves access to existing data; the additional index makes newly offloaded data more efficient to read. Existing objects do not acquire the new index automatically.
The implementation supports Amazon S3, Google Cloud Storage, and Azure Blob Storage. Applications continue reading through their existing Pulsar interfaces.
Test results: six subscriptions catching up together
An end-to-end test compared the previous tiered storage implementation with the new implementation. Both runs used the same workload configuration:
- One topic with one partition and 1 KiB messages.
- Six subscriptions, with one consumer per subscription.
- A configured producer rate of 10,000 messages per second.
- Consumers paused to build a backlog and allow offloading.
- Subscription positions staggered by 10,240 messages, followed by a topic unload before consumption resumed.
We calculated the results from 168 retained consumption-rate samples per run, using the arithmetic mean and median without removing outliers:
| Aggregate consumption throughput | Previous offload implementation | Tiered Storage v2 |
|---|---|---|
| Average | 12.4k msg/s | 76.6k msg/s |
| Median | 12.6k msg/s | 78.5k msg/s |
| Average of the final 10 samples | 12.6k msg/s | 74.7k msg/s |
The new implementation achieved approximately 6× the previous offload implementation's average aggregate consumption throughput in this test. These rates count deliveries across all six subscriptions, so the same published message can contribute more than once. They are not unique-message ingress rates or object-storage bandwidth.
Figure 2. Aggregate consumption throughput during catch-up. The chart displays the retained samples without smoothing.
The improvement remains visible in the final ten samples, where average throughput was approximately 74.7k msg/s versus 12.6k msg/s. This supports the benefit for concurrent catch-up beyond the overall average.
These results come from one retained comparison, with matching workload configurations but some variation in observed publishing rates. Complete infrastructure specifications and repeated-run measurements are unavailable. The test measures aggregate consumer throughput for a topic containing offloaded data; it does not isolate object-storage reads or the individual contributions of asynchronous execution, indexing, caching, and read-ahead, and it does not measure individual read latency or the time to drain the entire backlog. Actual gains depend on the workload, cache reuse, and available resources.
Production observations around a Tiered Storage v2 upgrade
The dashboards below show the same production deployment before and after its upgrade to StreamNative Tiered Storage v2, on the morning of July 22. They compare tiered-storage read bandwidth and message dispatch rate during ongoing production traffic. Each colored line represents an individual broker.
Figure 3. Reported tiered-storage read bandwidth drops sharply around the morning of July 22 and remains lower afterward. This visible change indicates the approximate upgrade period; it is not a deployment-log timestamp.
Figure 4. Message dispatch subsequently reaches a higher baseline following the visible transition in the same observation window, while spikes remain.
Read bandwidth drops first, while message dispatch subsequently settles at a higher baseline. The two changes occur at different points in the observation window. Taken together, they are consistent with the goal of reducing redundant object-storage reads, but the screenshots alone cannot distinguish the effects of the new reader from workload variation or conditions during the upgrade.
These production observations complement the six-subscription catch-up test above. The test measured approximately 6× the previous offload implementation's average aggregate consumption throughput, while the production dashboards show lower tiered-storage read bandwidth alongside a higher message-dispatch baseline.
Get Started
Ready to improve cold-data consumption with StreamNative Tiered Storage v2? If you’re an existing StreamNative customer, contact your account manager to upgrade for free. New to StreamNative? Sign up for free to get started.
