Amazon EventBridge connector is now integrated with StreamNative Cloud

What is Amazon EventBridge?

EventBridge is a serverless service that uses events to connect application components together, making it easier for you to build scalable event-driven applications. It can be used to route events from sources such as home-grown applications, AWS services, and third-party software to consumer applications across organizations. EventBridge provides a simple and consistent way to ingest, filter, transform, and deliver events to build new applications quickly.

What is StreamNative Cloud?

StreamNative Cloud is a fully managed cloud-native messaging and event streaming service built on Apache Pulsar.

‍

Why integrate StreamNative Cloud with Amazon EventBridge?

‍

StreamNative Cloud and Amazon EventBridge have different ecosystems and capabilities.

Synchronizing StreamNative Cloud data to EventBridge can help teams quickly access the AWS ecosystem, such as Lambda function, API Gateway, etc. Likewise, synchronizing Amazon EventBridge data to StreamNative Cloud can take advantage of more features of the Pulsar ecosystem, such as functions, data order guarantees, and flexible publish and subscribe models.

To integrate StreamNative Cloud with Amazon EventBridge, there are two data flow directions:

• StreamNative Cloud → EventBridge: Specify a topic in Pulsar, and whenever there is data in that topic, it is synchronized to an event bus in AWS EventBridge.
• EventBridge → StreamNative Cloud: Specify an event bus in AWS EventBridge, and whenever there is data in that event bus, it is synchronized to a topic in Pulsar.

‍

Here's how to implement each scenario.

StreamNative cloud data to AWS EventBridge.

This scenario can be implemented using a pulsar sink connector. StreamNative open-sourced the AWS EventBridge sink connector and has already integrated it into StreamNative Cloud. This connector can be used to easily send StreamNative Cloud messages to AWS EventBridge.

Create an AWS EventBridge sink connector on the StreamNative Cloud console.  

For more configuration information, please refer to the StreamNative Hub documentation.

‍

Tutorial: AWS EventBridge data to StreamNative cloud.

Since AWS EventBridge does not provide a similar poll/receive interface, this means that we cannot provide source connectors for StreamNative Cloud. AWS EventBridge uses API destinations to send msg from EventBridge to the third-party app. 

The StreamNative REST API can be configured in AWS EventBridge API destinations to send events to StreamNative Cloud.

> Note: If you use native Pulsar without StreamNative Cloud, you can’t integrate EventBridge into Pulsar data flow. Because the native Pulsar REST API does not support configured JSON content, StreamNative Cloud REST API improves it.

1. Get StreamNative Cloud REST config information

To get StreamNative Cloud REST Config:

1. Log in to StreamNative Cloud.
2. In the left navigation pane, choose Pulsar Clients.
3. Select Rest API table.
4. Select a service account.
5. Select the authentication type, and use Oauth2 and DownloadKey. Extract type, client_id, client_secret, client_email, and issuer_url from this key.
6. Show Client configuration. Extract grant_type, audience, and get_token_url from this information.
7. Select a topic and get Producer messages curl cmd. Extract produce_message_url from this cmd.

2. Create a connection on AWS EventBridge API destinations

> For details on how to create a connection, please refer to the official website.

According to the StreamNative Cloud REST config obtained in step-1, please fill it in as shown in the figure.

‍

When you create, you can see that the connection will start to authorize, and when the authentication is passed, the instructions are configured without problems.

‍

3. Create API destinations on AWS EventBridge 

> For details on how to create an API destination, please refer to the official website.

To create API destinations, you only need to fill in the produce_message_url obtained in step-1 to the endpoint and select the connection you just created.

4. Create a Rule on EventBus.

> For more on how to create rules, please refer to the documentation.

Create a Rule whenever the data in the EventBus conforms to the rule pattern, and send that data to API destinations.

In the target step, select the API destinations you created.

‍

5. Send events to EventBus.

You can simulate sending an event to the event bus to trigger it to SN Cloud.

‍

‍

6. Consume messages from StreamNative Cloud

‍

You can use client lib or pulsarctl to consume that topic message. 

The data that AWS delivers is a JSON, the REST API of StreamNative Cloud just uses byte schema, and you need to convert bytes to string. Its format is as follows:

{
	"version": "0",
	"id": "0dda1223-b6d0-0048-d75e-3aac6b5a3cd0",
	"detail-type": "com.test.People",
	"source": "api-destination-pulsar-rule",
	"account": "598203581484",
	"time": "2023-05-18T09:46:29Z",
	"region": "ap-northeast-1",
	"resources": [],
	"detail": {
		"name": "test-sn",
		"age": 20
	}
}

‍

The first layer of data format is fixed (detail-type, detail, etc), and the user's data is in the detail field.

In general, detail-type should contain type information about detail, and you can deserialize the content of detail based on detail-type. It depends on how the source of your data is set.

‍