Tow pages of the witepaper "Powering Federated Learning at Tencent with Apache Pulsar"

Powering Federated Learning at Tencent with Apache Pulsar

Tencent Angel PowerFL is a distributed federated learning platform which can support trillions of concurrent training. Angel PowerFL has been widely used in Tencent Financial Cloud, Advertising Joint Modeling, and other businesses. The platform requires a stable and reliable messaging system with guaranteed high performance and data privacy. After investigating different solutions and comparing several messaging queues, Angel PowerFL adopted Apache Pulsar as the data synchronization solution in Federated Learning (FL).

In this blog, the Tencent Angel PowerFL team shares how they built federated communication based on Pulsar, the challenges they encountered with Pulsar, and how they solved those problems and contributed to the Pulsar community. Tencent’s use of Pulsar in production has demonstrated it provides the stability, reliability, and scalability that the machine learning platform requires.

Picture of Chao Zhang from streamnative
Chao Zhang
Chao Zhang is a senior software engineer at the Tencent Data Platform Department. He specializes in Angel PowerFL Federated Communication / PowerFL on K8S. Together with the MQ team of the Tencent Data Platform Department, he introduces Apache Pulsar to the Angel PowerFL platform and starts new chapters for Pulsar adoption in machine learning.

Newsletter

The latest content on data streaming, delivered right to your inbox

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.