Apache Pulsar Adoption Story in ActorCloud (IoT Platform)

head img
Rocky Jin
September 9, 2019

Background

EMQ is an open-source software company providing highly-scalable, real-time messaging and streaming engine for IoT platforms & applications in the 5G era. Currently, EMQ is one of the most widely used MQTT message brokers in the world and has successfully supported various global clients, including HPE, Ericsson, Huawei, China Mobile, China UnionPay, and so on.

ActorCloud is an open-source IoT platform launched by EMQ, which provides multiple protocol access, message flow management, data parsing, and data processing capabilities for devices on a secure and reliable basis. ActorCloud uses Apache Pulsar to store and process streaming data, leverages Apache Pulsar Functions to handle data faster and analyzes IoT data through the SQL engine exposed to the upper layer.

Problem

As an IoT platform, ActorCloud needs to access data, manage devices, store and analyze data, and provide programming interfaces to the upper layer so that developers can develop applications conveniently. Since ActorCloud has plenty of devices and large amounts of data, it needs the ability to scale out horizontally to meet business needs.

Why Pulsar fits best

To solve the problem stated previously, we need a highly available, distributed, and scalable messaging platform, which is Apache Pulsar.

With the highly available and scalable ability, Functions, and connectors, Pulsar helps us develop ActorCloud faster so that we select Pulsar as our messaging platform finally.

How we use Pulsar at ActorCloud

ActorCloud transfers business logic written in SQL to an engine through API and translate business rules to connectors and Functions in Pulsar. Sources consume the data from EMQ X Brokers through shared subscriptions, then Pulsar persists these data and processes them with Functions in real time, and sends them to external systems through sinks.

Example

  
	{ 
		"id": "mailTest",
		"sql": "SELECT temp FROM sensor WHERE temp > 0",
		"enabled": true,
		"actions": [{
			"mail": {
				"title": "temperature warning",
				"content": "temperature is ${temp} degrees, please take action promptly",
				"emails": [ "alert@emqx.io" ]
			}
		}]
	}
    

How we use Pulsar Functions at ActorCloud

Apache Pulsar provides native support for serverless functions where data is processed as soon as it arrives in a streaming fashion and gives flexible deployment options (thread, process, container). We need only focus on computation logic rather than dealing with complicated configuration or management, which helps us build a streaming platform faster and conveniently.

How we use Pulsar IO connectors at ActorCloud

Pulsar IO connectors come in two types:

ActorCloud takes full advantage of Pulsar IO connectors and creates various sources and sinks to meet different needs.

  1. EMQ source

    Read data from EMQ and write data to Pulsar topics (sync data from EMQ with Pulsar).

  2. Mail sink

    Receive data from Pulsar topics and send emails.

  3. Publish sink

    Receive data from Pulsar topics and send data to external systems using initialized HttpClient.

  4. DB sink

    Receive data from Pulsar topics and send data to external systems. DB sink encapsulates JDBC and supports sending data from Pulsar topics to SQLite, MySQL, and PostgreSQL.

  5. EMQ sink

    Receive data from Pulsar topics and send data to EMQ X topics.

Summary

With both EMQ X and Apache Pulsar, ActorCloud implements IoT device data access, device management, data storage, data analysis, and provides a flexible programming interface to develop IoT applications that meet specific needs of IoT vertical industries, and it enables horizontal expansion of device access and data processing.

Do you like this blog? Share it now

twitter linkedIn