Driving Logistics Innovation: How Transport Exchange Group Modernized with Apache Pulsar and StreamNative

Transport Exchange Group (TEG) is a 24-year-old UK-based company operating in the logistics industry. The company’s exchange platform connects businesses with goods to deliver and carriers with available capacity. 

Recently, TEG expanded into finance, identity, and compliance services for the logistics sector. TEG has a diverse membership of goods distributors and carriers and has become an integral part of the UK's supply chain and logistics industry, with long-term plans for geographic expansion.

Challenge

As the company achieved growth and maturity, TEG needed to modernize its architecture, transitioning from a large monolithic model to an event-driven microservices architecture. With a team of 30 data, engineering, and DevOps employees, TEG required a scalable, high-performance messaging system that could handle real-time data processing, support transactional semantics, and integrate with various data sources including telematics providers and transport management systems. The solution needed to accommodate their complex algorithm for matching suitable carriers to available loads, which creates 0.5 million notifications per day, each within seconds of a load being posted on the platform.

Solution

After evaluating options, TEG chose Apache Pulsar, implemented through StreamNative's managed service. This decision was driven by Pulsar's support for transactional semantics, superior scalability without disruptive rebalancing, and its ability to handle elastic workloads. TEG adopted a hybrid approach, using StreamNative's hosted solution for non-production environments and a bring-your-own-cloud (BYOC) model for production workloads, optimizing for both performance and cost-effectiveness.

Technical Journey

TEG's migration to Pulsar began two years ago when they were using Kafka and looking to stream data from PostgreSQL. Pulsar's transactional capabilities and performance characteristics were key factors in the decision. TEG maintained some Kafka implementations for specific use cases while adopting Pulsar for new ones. They are now considering consolidating to Pulsar entirely as they move towards cloud environments, leveraging StreamNative Ursa engine’s Kafka compatibility capabilities.

Results

Since implementing StreamNative's Pulsar solution, TEG has experienced highly reliable service with only one production issue in two years, which was reported proactively with 24/7 monitoring by the StreamNative support team. Darren Parsons, Director of Architecture and Engineering, regards the platform as having been "bullet-proof," handling TEG's varying workloads efficiently, including demanding and busy seasons like the pre-Christmas period and other high-volume events such as bank holidays and major sporting events.

Key Takeaways

The adoption of StreamNative has provided TEG with a robust, scalable messaging system that meets their complex requirements. The company benefited from excellent support during onboarding and ongoing operations. TEG's experience highlights the importance of choosing a flexible, scalable solution that can grow with the business and adapt to changing needs.

Future Prospects

TEG is considering further optimization of their infrastructure, potentially consolidating their messaging platforms by leveraging Pulsar's Kafka protocol support. They continue to evolve their cloud architecture and may explore additional StreamNative offerings to simplify their overall system architecture. The company's focus on cost optimization and performance improvement suggests ongoing refinement of their Pulsar implementation.

Conclusion

StreamNative has enabled TEG to modernize its data streaming capabilities, supporting their transition to a more flexible and scalable architecture. The reliability, performance, and support provided by StreamNative have been crucial in maintaining TEG's operations across various peak periods and handling their dynamic workload requirements. As TEG continues to grow and expand globally, their Pulsar implementation provides a solid foundation for future developments.

"Our experience with StreamNative has been really good. From a service provider point of view, it's been bullet-proof. When we did have an issue, StreamNative knew about it before we did and had it fixed by the time we got into the office. That was a really positive experience." - Darren, Director of Architecture and Engineering, Transport Exchange Group

‍