Reducing Total Cost of Ownership (TCO) for Enterprise Data Streaming and Messaging

Modern enterprises are often faced with a conundrum: How can they leverage robust data streaming capabilities while keeping TCO within manageable bounds?

Conventional technologies such as Kafka or RabbitMQ pose significant challenges and expenses due to inefficient resource utilization resulting from their lack of elasticity, high infrastructure and operational costs from the multiplication of clusters through the organization, and the potential for downtime due to the absence of geo-replication. We introduce Apache Pulsar which natively solves those challenges with unique architecture and features and emerges as a transformative step towards a modern, scalable, and cost-effective data platform.

The Cost Impact of Traditional Data Streaming Technologies

Enterprises are increasingly reliant on data, generating vast quantities every second. Consequently, data streaming and messaging systems have emerged as critical components within these organizations, forming the bedrock of their ability to build real-time applications and derive actionable insights from their data.

‍
However, the decision to implement such systems requires an in-depth understanding of the total investment needed to produce the expected results. Those are not simply upfront expenses tied to setting up a new system. It encapsulates a wide range of factors, including the costs of installation, operation, maintenance, and upgrades. Even the potential financial implications of system downtime need to be accounted for in a comprehensive TCO analysis:

‍

• Infrastructure Costs: the cost of the resources you need to buy or rent to run your systems. Not all technologies are equally efficient when it come to using them. Traditional technologies like Kafka or RabbitMQ are typically unable to dynamically scale resources based on real-time demand. To ensure uninterrupted service, these platforms are often provisioned based on peak load estimates, with an additional buffer for unexpected surges. Consequently, during periods of lower demand, these systems are underutilized leading to inefficient resource utilization and higher overall server costs.
  

Also, more often than not, Enterprises simultaneously require messaging and streaming capabilities which lead to multiple clusters with independent resources,  piling up server costs (and operational costs), as well as data duplication, which inflates storage costs and complicates data management. 

• Operational Costs: The personnel, time, and financial resources devoted to tasks like ongoing monitoring, managing system updates, troubleshooting performance issues, and adjusting configurations for optimal data flow and security. For large-scale enterprises, these tasks can quickly balloon, especially when different applications or departments maintain separate clusters, which results in multiple instances of oversized hardware procurement, repeated setup, and maintenance tasks.

‍

A solution that provides centralized data governance and infrastructure management can drastically reduce these expenses, improving both cost-effectiveness and operational efficiency.

Similarly, managing different technologies for streaming and messaging use cases requires separate operational oversight (system updates, performance tuning, and troubleshooting) resulting in a higher expenditure of time, effort, and personnel resources with different skills. 

• Business Costs: traditional data streaming systems are often not built with robust fault tolerance and geo-redundancy mechanisms. Without such fail-safe measures in place, system disruptions, whether they arise typically from hardware failures, network issues, or human errors, can lead to significant data loss. Recovering from such incidents involves more than just the technical tasks of data restoration and system repair but also accounts for the 'business downtime' costs. These can include the loss of business opportunities during the outage, reputational damage, potential regulatory fines for data loss, and the loss of customer trust.

Finally, enterprises that choose to manage their own data streaming systems may experience significant opportunity costs. By allocating highly skilled personnel to these tasks, they divert valuable resources away from core business operations and strategic initiatives.

It's clear that while traditional data streaming technologies play a crucial role in businesses, they also come with a hefty price tag. 

Understanding Apache Pulsar

Originally developed by Yahoo with the intention to unify the best features of existing messaging systems in a cloud-native architecture, Apache Pulsar is now one of the most active projects of the Apache Software Foundation and offers a compelling blend of flexibility, scalability, and reliability. 

So, what makes Apache Pulsar unique?

In many traditional data streaming systems, serving and storage functions are intertwined. This architecture makes scaling a challenge as the growth in data volumes requires scaling both the serving and storage capacities concurrently, which is not optimal or cost-effective. At the opposite, Apache Pulsar's architecture fundamentally separates the serving and storage layers, which enables independent scaling of each. For example, if there's a spike in data intake, you can add more serving brokers without having to invest in additional storage capacity, and vice versa. This separation makes Pulsar highly elastic, allowing it to efficiently respond to changing data loads and throughput requirements. This architecture also forms the foundation for tiered storage, which allows for older data that are infrequently accessed to be offloaded from the serving brokers to cheaper, long-term storage, like Amazon S3. This significantly reduces storage costs and enables efficient utilization of more expensive primary storage.

On top of this great design, the community added great features, including:

• Multi-tenancy support,  which allows multiple teams or applications to share a single Pulsar cluster, effectively isolating their data and traffic. This means you can run multiple applications on a single Pulsar cluster with a high degree of security and isolation, thus maximizing the utilization of your resources.

• built-in support for geo-replication. Data can be served from local brokers for low latency access while being stored across different geographical regions for higher fault tolerance. In the event of a failure, Pulsar can seamlessly switch to brokers in a different region, minimizing downtime and potential data loss.
• Support of queuing and streaming use cases. This means you don't need separate systems for handling real-time and delayed data, simplifying your infrastructure and reducing maintenance overhead.

Furthermore, internal optimizations lead to better performance, and guaranteed message delivery, even in the face of network splits or server crashes. 

Why Apache Pulsar is cheaper

In a context of higher cost scrutinity, saving costs free up resources for other strategic initiatives, enhancing the overall efficiency and competitive edge of the organization. Apache Pulsar offers several key advantages to save costs: 

1. More efficient use of the infrastructure: Benchmarks show that Apache Pulsar has better raw performance and needs less hardware than Apache Kafka for a given throughput, partly due to the separation of the serving and storage layers that allows for separate cost optimization. The difference can go from 20% to 70% on a single cluster for the most demanding applications. 
2. Being elastic avoids oversizing: Apache Pulsar's elastic architecture allows you to scale your system dynamically according to your needs. Rather than requiring a significant upfront investment in high-specification machines, you have the flexibility to add more servers to your existing cluster as and when the data load increases. This flexibility ensures efficient resource allocation, reduces initial costs, and allows for better adaptability to changing business requirements. It also reduces the need for frequent maintenance and tuning when workload spikes, which in turn lowers operational costs.
3. Reduced costs for long retention: With Pulsar's tiered storage feature, old data can be offloaded from expensive primary storage to cheaper, long-term storage, reducing storage costs. 
4. Reduced downtime: Pulsar's built-in fault-tolerance and geo-replication features ensure data is securely backed up in multiple locations. This minimizes the risk of data loss or system outages, saving costs associated with downtime and disaster recovery. Its ability to seamlessly handle hardware failures or network splits without interrupting service is a critical advantage for businesses where every second of downtime translates into substantial financial losses.
5. Elimination of redundant systems: By offering a unified platform for both queuing and streaming, Apache Pulsar eliminates the need for maintaining separate systems for different types of data processing. This simplification not only reduces the costs associated with managing multiple systems but also lowers the risk of data inconsistencies and redundancies that can arise from using disparate systems. Moreover, the community and StreamNative are working on protocol handlers that allow existing applications, which currently rely on technologies like Kafka or RabbitMQ, to operate on Apache Pulsar without any modifications.. 
6. Resources and operational mutualization, thanks to multi-tenancy: As we discussed earlier, Apache Pulsar's native multi-tenancy support allows multiple applications or departments to share a single Pulsar cluster while maintaining strict data isolation. This leads to better resource utilization and lower hardware and maintenance costs, as the need for separate infrastructure for each application or department is eliminated.

In summary, 

• Apache Pulsar minimizes expenses on individual clusters through its superior performance and elasticity, eliminating the need for over-provisioning resources, and thereby avoiding underutilized infrastructure.
• When deployed as a shared platform across various teams within an organization, Apache Pulsar can lead to massive cost savings, as it allows efficient resource sharing and management.

Apache Pulsar presents a highly compelling option for enterprises seeking to enhance the efficiency and cost-effectiveness of their data streaming and messaging systems.  

Case Study: Orange Financial

Orange Financial is a major player in the mobile payment market with over 500 million registered users and 41.9 million active users, processing over 50 million transactions daily​​.

Prior to adopting Apache Pulsar, Orange Financial utilized a Lambda Architecture to handle its data processing needs, which involved splitting business logic into many segments and duplicating data across different systems for processing. This approach proved to be complex, hard to maintain, and costly. As their business grew, maintaining the different software stacks and clusters (including Kafka, Hive, Spark, Flink, and HBase) became prohibitively expensive​​.

Apache Pulsar was chosen to streamline their data processing stack, aiming to simplify the architecture, improve production efficiency, and reduce costs. With Pulsar, Orange Financial was able to unify log storage and computation into a single system, handling both real-time event streaming and processing. This resulted in a more robust and unified data serving layer​​.

The migration to Apache Pulsar also led to notable improvements in throughput and latency, allowing Orange Financial to handle peak traffic more effectively. The Pulsar-based system demonstrated high performance, being capable of responding to a transaction within 200 milliseconds​​.

Additionally, the use of Apache Pulsar's geo-replication and disaster recovery features resulted in a significant reduction of risk. By allowing data to be stored across multiple geographical locations, Orange Financial could ensure data availability and durability, even in the event of system failures​.

Ultimately, the switch to Apache Pulsar led to significant cost reductions. By simplifying their data processing architecture and reducing the need for maintaining multiple software stacks and clusters, Orange Financial was able to lower their operation and maintenance costs.

Apache Pulsar: A Strategic Investment

Considering Apache Pulsar for your organization's data streaming and messaging requirements is more than just a cost-saving decision. It represents a strategic investment toward improved data management and future readiness.

As data volumes continue to surge in the era of big data and real-time analytics, businesses need a robust, scalable, and reliable system that can keep pace. Apache Pulsar fits this requirement perfectly. Its ability to handle high volumes of data with ease, coupled with features like multi-tenancy, fault tolerance and geo-replication, makes it a future-proof solution for businesses.

Moreover, Pulsar's unified model for both streaming and queuing can simplify your data infrastructure, eliminating the need for multiple systems to manage different data processing needs. This not only reduces the potential for data inconsistencies but also makes the system easier to manage and scale.

Apache Pulsar also encourages better resource utilization through its multi-tenancy support. This feature is especially beneficial for enterprises where multiple departments or applications may need access to data streaming and messaging services. Instead of investing in isolated clusters for each application or department, businesses can consolidate their resources, leading to substantial cost savings and improved operational efficiency.

Given these advantages, investing in Apache Pulsar can be seen as a strategic move towards modern, efficient, and cost-effective data management. 

Conclusion

In a world where data is increasingly becoming the lifeblood of organizations, choosing the right data streaming and messaging system can significantly influence a company's efficiency, responsiveness, and competitive edge. The total cost of ownership is a crucial factor that can't be overlooked when evaluating these systems.

Case studies of global organizations show that the shift to Apache Pulsar can lead to substantial cost savings, improved operational efficiency, and enhanced system stability. By integrating Apache Pulsar into their technical infrastructure, these organizations have made a strategic investment toward future-proofing their data management systems. While the transition may entail certain costs and challenges in the short term, the long-term benefits in terms of greatly reduced costs and improved operational efficiency are well worth the effort.

StreamNative has helped engineering teams worldwide make the move to Pulsar. Founded by the original creators of Apache Pulsar, StreamNative is one of the leading contributors to the open-source Apache Pulsar project and the author of the StreamNative Operators for running Apache Pulsar on Kubernetes, and of StreamNative Cloud, a fully managed service to help teams accelerate time-to-production.

‍