Developing Event Driven Microservices with Apache Pulsar: Part I

Our Developer Advocate team recently presented a 3-part webinar series on building event-driven microservices with Apache Pulsar. To build on that topic, we will be publishing a blog series and this is the first blog in that series.

In this blog, you will learn how Apache Pulsar’s support for common message patterns and native compute capabilities, known as Pulsar Functions, can be leveraged to build a message bus for event driven microservices. Additionally, you will learn how to use Pulsar Functions to create lean, testable event driven microservices for a diverse set of deployment use cases.

Why Apache Pulsar for Event Driven Microservices

Event-driven microservices use a message bus to communicate among loosely-coupled, collaborating services. When a service performs work that other services might be interested in, that service produces an event. Other services can then consume the event in order to perform their own tasks.

The message bus serves as an intermediary between the different services. The message bus receives events from producers, filters the events, then pushes the events to consumers without tying the events to individual services. In order to accomplish this, the message bus needs to support various messaging paradigms, subscription types, and consumption patterns.

Apache Pulsar is a cloud-native, distributed messaging and event-streaming platform that supports common message patterns with its diverse subscription types and modes. The ability to support a diverse number of messaging patterns is important because, as mentioned above, it is required for many types of microservices.

In addition, Pulsar includes native, lightweight compute capabilities known as Pulsar Functions that allow you to build microservices with a few lines of code. You can write Pulsar Functions in Java, Python, or Golang and deploy them in threads, processes, or Kubernetes pods. (We will talk more about Pulsar Functions later in this blog.)

Pulsar also provides scalability and elasticity, which are critical to microservices. Pulsar has a cloud-native, layered architecture that separates compute and storage into different layers. Decoupled compute and storage allows for independent scaling and enables microservices to scale elastically on short notice.

Furthermore, microservices built with Pulsar can easily integrate with external frameworks, libraries, and systems. Pulsar has many connectors, such as MongoDB, ElasticSearch, Aerospike, InfluxDB, and Redis, and protocol handlers, including JMS, AMQP, and MQTT. You can explore what is available in the Pulsar ecosystem in the StreamNative Hub.

Building Event-Driven Microservices with Pulsar Functions

In the previous section, we discussed why Apache Pulsar is well-suited for event-driven microservices. In this section we cover why you should use Pulsar Functions when developing microservices.

Pulsar Functions are lambda-style functions and make it easy to transform or process messages in Pulsar. The diagram below illustrates the programming model of Pulsar Functions.

Pulsar Functions complete the following tasks when input messages are received:

• Consume messages from one or more Pulsar topics
• Apply a user-supplied processing logic to each message
• Publish the results of the computation to another topic
• Write logs to a log topic (potentially for debugging purposes)

When you use Pulsar Functions, producers and consumers are automatically set up, removing the need to write boilerplate code. When messages arrive in a topic, Pulsar Functions automatically applies user-supplied business logic components and sends the output to the specified topics.

The code below is an example of a microservice written as Pulsar Functions. The API is simple, allowing developers to focus on the business logic and easily write services, without the need for deep streaming knowledge. Because of this, Pulsar Functions can easily be developed by a small team or a single developer, allowing agile teams to build microservices.

<script>
import java.util.function.Function;

public class NVSalesTaxCalcService implements Function {
    public String apply(Float taxableAmt) {
       return taxableAmt * 0.08125;   // 8.125% sales tax
    }
}
<script> 

You can use Pulsar’s native function capabilities for many use cases, including simple calculations, filtering, and single message transformation. However, if you need to develop complex functions or access Pulsar’s metadata, you can use the Pulsar client library to integrate with a variety of systems, such as the machine learning library TensorFlow.

Benefits of Pulsar Functions

Now that we know how Pulsar Functions work, we can look at the key benefits of using Pulsar Functions to build event-driven microservices.

1. Flexible Deployment Options

Microservices written as Pulsar Functions can be deployed individually as threads, processes, or Kubernetes Pods inside Pulsar. The functions are not tied to each other at compile, run, or deployment time and there is no need for another processing framework, deployment system, FaaS, or specialized server. Since the microservices are isolated, you can scale each independently to support any workload.

2. Function Mesh for Complex Operations

For more advanced deployment, you can use a tool called Function Mesh to deploy multiple Pulsar Functions as a single unit. Function Mesh allows developers to utilize the full power of Kubernetes Scheduler with Pulsar Functions, including deployment, scaling and management. To learn more about Function Mesh, you can read this blog.

3. Maintainability and Testability

Pulsar Functions are highly maintainable and testable, which are desired quality for microservices. They are small pieces of code written in popular languages, such as Java, Python, or Go. They can be easily maintained in source control repositories and automatically tested with existing frameworks.

Conclusion

In this blog, we explained why you should leverage Apache Pulsar to build event-driven microservices. Pulsar enables teams of all sizes to build scalable, maintainable, testable microservices that support flexible deployment. Its simple API allows developers to focus on the business logic without the need for deep streaming knowledge.

In the next blog in this series, we will show you:

• Example code of building microservices with Pulsar Functions.
• How to design schemas using the Avro interface definition language and use the schemas in Pulsar Functions-based microservices.
• How to deploy microservices to a Pulsar cluster.

More on Pulsar

1. Learn the Pulsar Fundamentals: While this blog did not cover the Pulsar fundamentals, there are great resources available to help you learn more. If you are new to Pulsar, we recommend you to take the self-paced Pulsar courses or instructor-led Pulsar training developed by some of the original creators of Pulsar. This will get you started with Pulsar and accelerate your streaming immediately.

2. Spin up a Pulsar Cluster in Minutes: If you want to try building microservices without having to set up a Pulsar cluster yourself, sign up for StreamNative Cloud today. StreamNative Cloud is the simple, fast, and cost-effective way to run Pulsar in the public cloud. You can spin up a Pulsar cluster in minutes.

3. Continued Learning: If you are interested in learning more about microservices and Pulsar, take a look at the following resources:

• [3-Part Webinar Series] Building Event-Driven Microservices with Apache Pulsar. Watch the webinars here and find the source code from the webinars here.
• [Doc] How to develop Pulsar Functions
• [Blog] Function Mesh - Simplify Complex Streaming Jobs in Cloud