Unlocking Real-Time Data Synergy: TiDB Cloud Changefeed Integrates with StreamNative Cloud

We are excited to announce a strategic collaboration between TiDB Cloud and StreamNative, enabling you to stream real-time data from TiDB Cloud directly into the StreamNative Cloud ecosystem. This integration utilizes the new changefeed feature in TiDB Cloud to capture data changes as they occur and seamlessly stream them to topics hosted on StreamNative Cloud.

Unlocking Real-Time Data Synergy

In today's fast-paced digital landscape, the ability to process and react to data in real-time is essential. TiDB Cloud offers a powerful, distributed SQL database that excels at handling large-scale, high-concurrency workloads. StreamNative's ONE platform, powered by Apache Pulsar, on the other hand, provides a highly scalable, low-latency messaging and streaming platform designed for real-time data ingestion and distribution. The new changefeed feature bridges these two powerful platforms, enabling seamless data flow from TiDB Cloud to StreamNative Cloud.

This integration allows users to capture data changes in TiDB Cloud and stream them directly into Apache Pulsar topics on StreamNative Cloud. This opens up a multitude of use cases, including:

• Real-time Analytics: Feed operational data from TiDB Cloud into Pulsar for immediate processing and analytical insights.
• Event-Driven Applications: Build reactive applications that respond instantly to database changes.
• Data Synchronization: Keep various data systems consistent by propagating changes efficiently.
• Data Lake Ingestion: Stream TiDB Cloud data into data lakes for historical analysis and machine learning.

Getting Started with TiDB Cloud Changefeed and StreamNative Cloud

Getting started with this powerful integration is straightforward. Here's a quick guide to help you set up your first changefeed:

Prerequisites

• StreamNative Cloud account
• TiDB Cloud account
• Basic understanding of Apache Pulsar topics and subscriptions

Step 1: Prepare Your StreamNative Cloud Environment

Before configuring the changefeed, you need to set up the destination topic and get authentication details from your StreamNative Cloud console.

1. Log in to your StreamNative Cloud console. You can create a serverless cluster to get started quickly.
2. Create a Topic: Navigate to the Topics section, select your desired tenant and namespace, and create a new topic (e.g., tidb-changes). Note the full topic name, as you will need it later.
3. Get Authentication Key: Navigate to 'Pulsar Clients‘, select your service account, then click Create API Key to generate a token. Copy this token securely.

Step 2: Configure TiDB Cloud Changefeed

With your StreamNative Cloud access APIKey ready, you can now create the changefeed in TiDB Cloud.

1. Log in to your TiDB Cloud console.
2. Navigate to the Changefeed section and create a new changefeed.
3. Select Pulsar as the destination.
4. Under Connection, provide your Pulsar Broker URL and select the Token Auth Type.
5. Enter the broker Pulsar Broker URL and API key you obtained in the previous step.

Step 3: Create a Table and Insert Data in TiDB Cloud

Now that the changefeed is set up, you can generate change events by executing SQL commands in the TiDB Cloud Shell.

1. Open the TiDB Cloud Shell for your cluster.
2. Run the following SQL statements to create a users table and insert two records:

Step 4: Verify Data in StreamNative Cloud

The change events from the SQL commands will be streamed to your StringNative Apache Pulsar topic. You can use a Pulsar consumer to verify that the data has been received correctly.

The consumer will receive the change events in a structured JSON format. The first event will be the DDL for the CREATE TABLE command, followed by separate DML events for each INSERT operation.

For detailed instructions and advanced configurations, please refer to the official documentation:

• StreamNative Documentation
• TiDB Cloud Documentation

Empowering Developers and Enterprises

This collaboration between TiDB Cloud and StreamNative is a testament to our shared commitment to fostering open-source innovation and empowering our users with best-in-class data infrastructure. By seamlessly integrating TiDB Cloud's robust data capabilities with Apache Pulsar's real-time streaming prowess, we are providing a comprehensive solution for building next-generation, data-intensive applications.

We invite you to explore this new feature and experience the synergy of TiDB Cloud and StreamNative Cloud firsthand.

About StreamNative

StreamNative is the leading contributor to Apache Pulsar and the provider of StreamNative Cloud, a fully-managed Apache Pulsar service. StreamNative empowers organizations to build and deploy real-time applications at scale, leveraging the power of Apache Pulsar for event streaming, messaging, and queuing.

• Learn more about StreamNative Cloud
• Sign up for a trial to get $200 credit

About TiDB Cloud

TiDB Cloud is the company behind TiDB, the open-source, distributed SQL database that enables developers and DBAs to build scalable, resilient, and real-time applications. TiDB Cloud is TiDB Cloud's fully-managed database-as-a-service, simplifying database operations and allowing users to focus on innovation.

• Learn more about TiDB Cloud

Follow us for more updates

• StreamNative
• PingCAP