Transforming Indoor Analytics with StreamNative: InnerSpace's Journey to Scalable Real-Time Data Processing

Featuring: Matt MacGillivray, Co-Founder, Vice President of R&D at InnerSpace

Background

InnerSpace is changing how people experience the indoors by helping them understand how their spaces should be used. The company’s technology creates reliable and insightful data about people’s behavior to those seeking to implement solutions that improve indoor experiences. InnerSpace uses existing WiFi infrastructure to analyze anonymous signals in real-time from mobile devices and translate them into actionable and easy-to-understand insights about how and where we spend our time indoors.

Challenge

InnerSpace needed a robust streaming solution to handle their real-time and complex data processing needs. Initially, InnerSpace utilized regular queues and OpenFaaS function-as-a-service for ingesting raw data, including signal strengths from phones to determine locations. As they expanded their operations to include an incredibly high volume of data where they were function processing every 60-seconds, high-level processing, and transforming that data into actionable locations and insights, they recognized the need for a more sophisticated solution. 

Solution

InnerSpace began by adopting a Function as a Service product, but volume was a significant problem. The team could only conduct small deployments, cost was high, and it was complex with multiple stacks and queue providers. 

After conducting research, InnerSpace identified Apache Pulsar as the ideal fit for their requirements. They were initially attracted to the function and execution capabilities, with extremely low latency speeds. However, being a small company, they were hesitant to manage the complexity of Pulsar in-house. They wanted an easily managed solution that could be managed by one person, which led them to StreamNative, a fully managed Pulsar service that aligned with their needs for a comprehensive setting. Upfront, they were appreciative of the cost as it combined both their processing time and management. InnerSpace swiftly transitioned their operations to StreamNative, integrating it across all their use cases and benefiting from its managed infrastructure.

Technical Journey

InnerSpace's technical journey with StreamNative involved implementing a complex data pipeline that processed information through 6-8 different topics before reaching its final destination. 

As their needs evolved, InnerSpace began transitioning from a managed service to a cloud-based solution to leverage additional benefits. The data pipeline culminates with information being written to Snowflake, their main data warehouse, while also interacting with Databricks. This setup facilitates a continuous flow of data between these platforms, enabling data scientists to perform computations and push results back to Snowflake. The entire system is underpinned by block storage in Microsoft Azure, creating a robust and flexible infrastructure for InnerSpace's data processing needs. The cost is half that of their managed Pulsar deployment, with minimal DevOps involvement, which is ideal for their IT team.

Key Takeaways

As a startup company, InnerSpace reaped significant benefits from StreamNative's fully managed service, allowing them to harness advanced Pulsar technology without the burden of in-house management. This enabled InnerSpace to focus on their core business of indoor space analytics. 

The solution seamlessly integrated with InnerSpace's existing tech stack, including Snowflake and Databricks, creating a robust data pipeline that enhanced their analytical capabilities. StreamNative's adaptability shone through as it supported InnerSpace's transition to a cloud-based solution, demonstrating its ability to evolve with changing business needs.

We jumped into StreamNative fairly early after finding out about the power of Apache Pulsar, but not initially having the headcount to manage it ourselves. We wanted something that was full service that offered function processing and persistent topics. StreamNative is an excellent fit for all of our real-time processing use cases. It gave us functions that were executed quickly with extremely low latency.

Future Prospects

InnerSpace’s next goal is to continue to roll out vertical function scaling, addressing the fluctuating demands of office buildings, which experience high activity during the day and minimal usage at night. This feature allowed them to reduce CPU usage during off-peak hours, optimizing resource allocation. This, in turn, will help them with continued cost savings. 

Conclusion

StreamNative's Apache Pulsar-based platform proved to be a game-changer for InnerSpace, offering unparalleled scalability and flexibility to handle varying workloads efficiently.