One Platform, Two Profiles: Streaming for Latency or Cost

As streaming workloads continue to diversify, infrastructure requirements have become increasingly workload-specific. Low-latency, transactional event flows impose very different constraints than high-throughput ingestion pipelines feeding analytical systems or lakehouse storage.

StreamNative’s evolution—from the Classic engine to the Ursa engine—has been driven by these differing requirements. Cluster Profiles formalize this evolution by allowing customers to explicitly select the infrastructure characteristics that best match their workload: latency optimization or cost optimization.

StreamNative Classic Engine: Latency-First Architecture

The StreamNative Classic engine is based on the original Apache Pulsar architecture, built on ZooKeeper for metadata coordination and BookKeeper for durable log storage.

This design offers:

• Strong consistency and predictable write/read paths
  
• Tight coupling between compute and storage for low-latency access
  
• Mature operational semantics for real-time workloads
  
• Native support for both Pulsar and Kafka protocols
  

The Classic engine has been successfully deployed for latency-sensitive use cases where end-to-end response time and deterministic behavior are critical. However, this architecture inherently couples scaling and cost to persistent storage and network resources, making it less optimal for workloads prioritizing elastic scale and cost efficiency.

Ursa Engine: Cloud-Native, Storage-Decoupled Streaming

To address scale, cost, and operational efficiency at cloud scale, StreamNative introduced the Ursa engine, a next-generation streaming architecture designed around storage disaggregation.

Key architectural characteristics include:

• Object storage–based durability using S3, GCS, or Azure Blob Storage
  
• Oxia for scalable, fault-tolerant metadata management
  
• Decoupled compute and storage layers enabling independent scaling
  
• Reduced reliance on replicated block storage and cross-AZ networking
  

By shifting durability to object storage and re-architecting metadata handling, Ursa significantly lowers infrastructure costs while enabling high-throughput streaming at scale. The Ursa engine is the strategic foundation for StreamNative’s future innovation and supports Kafka and Pulsar APIs without requiring protocol-specific infrastructure.

Abstracting Engines with Cluster Profiles

While both engines serve valid and important use cases, exposing engine choice directly to customers adds unnecessary complexity. Most users care less about internal architecture and more about performance characteristics, cost models, and operational behavior.

Cluster Profiles provide a higher-level abstraction that maps workload intent to infrastructure behavior.

Cluster Profiles in Detail

Latency Optimized Cluster Profile

The Latency Optimized profile is designed for workloads that require consistently low end-to-end latency and predictable performance under load.

Technical characteristics:

• Block storage–backed persistence using Apache BookKeeper for durable, replicated log storage
• ZooKeeper-based metadata coordination for leader election and cluster state management
• Tightly coupled compute and storage optimized for low-latency read/write paths
• Synchronous replication to minimize tail latency and ensure predictable durability guarantees
• Optimized networking paths for fast broker-to-storage communication
• Native support for Pulsar and Kafka APIs with consistent semantics
  

Recommended workloads:

• Event-driven microservices where events synchronously trigger downstream services and user-facing actions
• Real-time transaction processing such as payments, order placement, and inventory updates with strict SLAs
• Fraud detection and risk scoring pipelines requiring immediate event evaluation before action is taken
• Operational alerting and monitoring systems with low tolerance for delivery latency or jitter
• Online personalization and recommendation triggers where user experience depends on sub-second responses
• Control-plane and coordination messaging for distributed systems requiring fast and consistent state propagation
  

Cost Optimized Cluster Profile

The Cost Optimized profile leverages the Ursa engine’s storage-disaggregated architecture to maximize efficiency at scale.

Technical characteristics:

• Object storage–backed persistence using Amazon S3, Google Cloud Storage, or Azure Blob Storage
• Oxia-based metadata management for scalable, fault-tolerant coordination
• Decoupled compute and storage layers enabling independent scaling and elasticity
• Asynchronous durability and batching optimized for throughput and cost efficiency
• Reduced reliance on replicated block storage and cross–availability zone networking
• Unified support for Pulsar and Kafka APIs without protocol-specific infrastructure

Recommended workloads:

• High-volume event ingestion from IoT devices, mobile applications, or telemetry sources
• Streaming pipelines feeding lakehouse platforms such as Apache Iceberg and Delta Lake for analytics and AI
• Clickstream and behavioral analytics optimized for throughput and scalable processing
• Long-term event retention and replay for compliance, auditing, or ML feature backfills
• Data replication and fan-out pipelines across regions, clouds, or downstream systems
• Batch-to-stream modernization workloads where elasticity and cost efficiency outweigh low-latency requirements

Cost Characteristics by Cluster Profile

StreamNative Cluster Profiles are designed to align infrastructure costs with workload requirements: the Latency Optimized profile prioritizes predictable, low-latency performance using tightly coupled compute and storage, resulting in higher infrastructure costs, while the Cost Optimized profile leverages a storage-disaggregated, object-storage-backed architecture to significantly reduce storage, replication, and operational costs for high-throughput workloads.

Unified Platform, Consistent APIs

Importantly, Cluster Profiles do not fragment the StreamNative platform. Across profiles, customers benefit from:

• A unified control plane and operational model
• Consistent Kafka and Pulsar APIs
• Shared security, governance, and observability capabilities
  

This allows teams to deploy multiple clusters with different profiles—aligned to workload requirements—without introducing platform sprawl.

Infrastructure Choice as a First-Class Concept

Cluster Profiles represent a shift from engine-centric thinking to workload-driven infrastructure selection. Instead of adapting applications to infrastructure constraints, customers can now select an infrastructure profile that aligns with latency, cost, and scale requirements—while remaining on a single, coherent streaming platform.

This approach reflects StreamNative’s broader strategy: evolving the platform architecture while simplifying how customers consume and operate streaming infrastructure.

Summary

StreamNative Cluster Profiles let you select streaming infrastructure based on what matters most—low latency or cost efficiency—building on StreamNative’s evolution from the Classic engine to the cloud-native Ursa engine. 

To get started with StreamNative Cloud, Sign up for a trial.