EMQ is an open-source software company providing highly-scalable, real-time messaging and streaming engine for IoT platforms & applications in the 5G era. Currently, EMQ is one of the most widely used MQTT message brokers in the world and has successfully supported various global clients, including HPE, Ericsson, Huawei, China Mobile, China UnionPay, and so on.
ActorCloud is an open-source IoT platform launched by EMQ, which provides multiple protocol access, message flow management, data parsing, and data processing capabilities for devices on a secure and reliable basis. ActorCloud uses Apache Pulsar to store and process streaming data, leverages Apache Pulsar Functions to handle data faster and analyzes IoT data through the SQL engine exposed to the upper layer.
As an IoT platform, ActorCloud needs to access data, manage devices, store and analyze data, and provide programming interfaces to the upper layer so that developers can develop applications conveniently. Since ActorCloud has plenty of devices and large amounts of data, it needs the ability to scale out horizontally to meet business needs.
To solve the problem stated previously, we need a highly available, distributed, and scalable messaging platform, which is Apache Pulsar.
With the highly available and scalable ability, Functions, and connectors, Pulsar helps us develop ActorCloud faster so that we select Pulsar as our messaging platform finally.
ActorCloud transfers business logic written in SQL to an engine through API and translate business rules to connectors and Functions in Pulsar. Sources consume the data from EMQ X Brokers through shared subscriptions, then Pulsar persists these data and processes them with Functions in real time, and sends them to external systems through sinks.
Example
{
"id": "mailTest",
"sql": "SELECT temp FROM sensor WHERE temp > 0",
"enabled": true,
"actions": [{
"mail": {
"title": "temperature warning",
"content": "temperature is ${temp} degrees, please take action promptly",
"emails": [ "alert@emqx.io" ]
}
}]
}
Apache Pulsar provides native support for serverless functions where data is processed as soon as it arrives in a streaming fashion and gives flexible deployment options (thread, process, container). We need only focus on computation logic rather than dealing with complicated configuration or management, which helps us build a streaming platform faster and conveniently.
Pulsar IO connectors come in two types:
ActorCloud takes full advantage of Pulsar IO connectors and creates various sources and sinks to meet different needs.
EMQ source
Read data from EMQ and write data to Pulsar topics (sync data from EMQ with Pulsar).
Mail sink
Receive data from Pulsar topics and send emails.
Publish sink
Receive data from Pulsar topics and send data to external systems using initialized HttpClient.
DB sink
Receive data from Pulsar topics and send data to external systems. DB sink encapsulates JDBC and supports sending data from Pulsar topics to SQLite, MySQL, and PostgreSQL.
EMQ sink
Receive data from Pulsar topics and send data to EMQ X topics.
With both EMQ X and Apache Pulsar, ActorCloud implements IoT device data access, device management, data storage, data analysis, and provides a flexible programming interface to develop IoT applications that meet specific needs of IoT vertical industries, and it enables horizontal expansion of device access and data processing.