How Q6 Cyber Tamed 85+ Billion Cyberthreat Records with Apache Pulsar & StreamNative

Background

Q6 Cyber is a leading provider of actionable threat intelligence, focusing specifically on financial fraud prevention. Founded in 2016, the company delivers intelligence that helps financial institutions identify and mitigate cyber threats before they result in financial losses. Unlike traditional threat intelligence companies that focus on general cybersecurity threats, Q6 Cyber collects data directly from the cybercriminal ecosystem—compromised credentials, malware command-and-control servers, and dark web forums—and processes this data into actionable, easy-to-consume intelligence that their clients can immediately utilize to prevent fraud.

Challenge

As Q6 Cyber's intelligence operations expanded to collect and process billions of threat intelligence records, they faced significant scaling challenges with their data infrastructure. 

The company’s Google Cloud Pub/Sub implementation struggled with throughput during high-volume data ingestion periods. With over 85 billion records collected, their primary OpenSearch cluster had become unwieldy to manage and was hitting performance limitations. Q6 Cyber needed a more flexible solution that could handle unpredictable volume spikes while supporting their plans to transition to a data lake architecture. 

Additionally, the sensitive nature of their data required complete control over their infrastructure, making a Bring Your Own Cloud (BYOC) solution essential.

Solution

Q6 Cyber implemented StreamNative's platform with Apache Pulsar at its core to serve as the central nervous system of their data processing architecture. The solution provided high-performance messaging with better throughput and reliability than their previous implementation, while the BYOC option allowed them to maintain complete control over their sensitive data. 

Pulsar's native schema management simplified data processing across diverse formats and sources, while Pulsar Functions enabled them to deploy lightweight processing logic directly within the messaging layer. Most importantly, StreamNative positioned Q6 Cyber to execute their planned migration of 85 billion records to a data lake architecture, providing the reliable transport layer needed for this massive undertaking.

Technical Journey

Q6 Cyber initially adopted open-source Apache Pulsar and later transitioned to StreamNative's managed platform with the BYOC option. They began by introducing Pulsar alongside Google Pub/Sub for specific high-throughput use cases, allowing them to validate the technology without disrupting existing workflows. 

As confidence in the platform grew, they shifted more of their data flows to Pulsar, particularly for new applications. They leveraged Pulsar's unified messaging model to simplify their architecture, transitioning from disparate systems to a more centralized approach with Pulsar at the core. With this foundation in place, Q6 Cyber began designing their data lake migration strategy, with StreamNative serving as the critical transport layer.

Results

StreamNative's platform delivered more consistent, reliable performance for Q6 Cyber's variable workloads, particularly during high-volume ingestion periods. The architectural flexibility allows them to easily route data to different systems as needed, with Pulsar serving as the center of their data architecture. 

Additional benefits include:

1. While Q6 Cyber's initial vision for using Pulsar focused on long-term tiered storage, the flexibility of the platform allowed them to adapt their architecture as their strategy evolved.
2. Having schema validation at the transport layer proved more valuable than initially anticipated, reducing development complexity and improving data quality.
3. The platform's ability to handle variable workloads proved essential for a threat intelligence provider dealing with unpredictable surges in cybercriminal activity.

Future Prospects

Q6 Cyber plans to complete their data lake migration, leveraging StreamNative to move tens of billions of records while maintaining operational continuity. They're also interested in further leveraging Pulsar Functions as improvements are made to support their high-fanout publishing scenarios, where a single input can generate numerous outputs. As they continue to scale their threat intelligence capabilities, StreamNative's platform will remain central to their data architecture.

Conclusion

StreamNative's platform has positioned Q6 Cyber to execute on their vision of a more scalable, flexible threat intelligence infrastructure. By providing a high-performance, reliable messaging layer with the security and control they require, StreamNative has enabled Q6 to focus on their core mission: delivering actionable threat intelligence to prevent financial fraud. As cyber threats continue to evolve, Q6 Cyber now has the infrastructure flexibility needed to adapt quickly, ensuring they can continue to protect financial institutions and their customers.