Pulsar Hackathon 2021 Winners Announced

2021 has been an exciting year for the Apache Pulsar community. In May the community hit the 400 Contributor mark and Apache Pulsar surpassed Apache Kafka in Monthly Active Contributors!

To continue the Pulsar momentum, StreamNative hosted the first-ever Pulsar Hackathon May 6th & 7th, 2021. The goal of the Hackathon was to engage the Pulsar community, drive contributions, and generate ideas to enhance Pulsar and its ecosystem, and, with more than 130 signups for the event, it was a success! We’d like to start by saying thank you to everyone who participated.

In this post we’ll share more on the winners and their projects. But before we get to the winners, let’s talk about the challenge.

The Categories

To help inspire the teams, we created five categories for the hackathon:

• Pulsar Enhancement: adding new features, improving performance, etc.
• Pulsar + Big Data Ecosystem Integration: integrating Pulsar with other influential data systems for easy usage
• Pulsar + Flink Solution: developing end-to-end general data processing solutions based on Pulsar and Apache Flink
• Pulsar + Cloud: enabling Pulsar to run on cloud environments easily and seamlessly
• BookKeeper Enhancement: adding new features, improving performance, etc.

The Judges

A panel of seven judges with deep experience in Apache Pulsar and real-time data streaming and messaging technologies was assembled. That panel included:

• Matteo Merli, Apache Pulsar Chair, CTO at StreamNative
• Jerry Peng, Apache Pulsar PMC Member, Principal Software Engineer at Splunk
• Addison Higham, Chief Architect at StreamNative
• Ricardo Ferreira, Principal Developer Advocate at Elastic
• Sijie Guo, Apache Pulsar PMC Member, CEO at StreamNative
• Nozomi Kurihara, Manager of the Messaging Platform Team at Yahoo! Japan
• Arvid Heise, Senior Engineer at Ververica

The teams were judged on three criteria: Innovation, Utility / Applicability, Difficulty.

The Winners

After two days of hard work, 11 teams submitted their projects by the deadline. The winners were announced live at the Pulsar Virtual Summit North America 2021 on June 16th and 17th. Below are the top-rated submissions.

First Place ($5,000 Prize): ZooKeeper Free

This project eliminates Pulsar’s dependency on ZooKeeper by bringing metadata storage and management into BookKeeper. This project also enables Pulsar users to handle metadata more flexibly by introducing a unified metadata API for brokers and bookies. Team members include Bo Cong, Ran Gao, Yang Yang, Yu Liu, and Zike Yang.

“It not only innovates Pulsar's architecture but also minimizes the operational complexity of it's architecture. It is by far the hardest project to implement.” - Ricardo Ferreira, Hackathon Judge

“Concise / Easy to understand.” - Nozomi Kurihara, Hackathon Judge

Watch the demo here.

Second Place ($2,500 Prize): Support Hyperscale Topics and Clients

This project enables scale in producers and topics by reducing load in ZooKeeper and introducing the Topic Level Batch approach. Team members include Lin Lin, Hang Chen, and Penghui Li.

“A simple idea to scale the production workload.”- Sijie Guo, Hackathon Judge

Watch the demo here.

Third Place ($1,000 Prize): Pulsar Watermarking

This project solves the challenge of generating watermarks when consuming a Pulsar topic. It teaches Pulsar how to broker the transmission of event time watermarks from producers to consumers. Team members include Jennifer Huang, Eron Wright, Giannis Polyzos, and Murthy Kakarlamudi.

“The overall idea of this project is interesting because out-of-order events are likely to happen in stream processing apps.” - Ricardo Ferreira, Hackathon Judge

“This is a really innovative feature that no other streaming system seems to have. ” - Addison Higham, Hackathon Judge

Watch the demo here.

Fourth Place: Pulsar Multi-Tenant BookKeeper Storage Isolation

This project teaches Pulsar to store entryLogFiles to different folders according to the tenant. Team members include Jialing Wang, Hao Zhang, Shaojie Wang, and Xin Yi.

“Super useful feature for making multi-tenancy easier to manage at the storage layer. This, paired with some improvements around quotas, would be very useful in helping to manage larger Pulsar clusters” - Addison Higham, Hackathon Judge

Watch the demo here.

Fifth Place: Integrated with APM

This project aims to fill in the missing tracing piece of Pulsar's Ops and achieve sample tracing in Pulsar Broker and integration with SkyWalking. Team members include Zhangjian He and Tian Luo.

“This use case is quite useful for troubleshooting end-2-end Pulsar applications.” - Ricardo Ferreira, Hackathon Judge

Watch the demo here.

The Innovation Continues: StreamNative Incubator Program

Hackathon participants will be able to continue their projects as part of the StreamNative Incubator Program. This 12-week engagement will pair each participating team with a mentor from the Pulsar/BookKeeper communities. Together, the team members and mentor will lay out a practical execution plan for the project and work together to execute the final code. We will share progress and updates with the Pulsar community!

Stay involved:

Join Apache Pulsar on Slack. Sign up for the StreamNative Newsletter.