Revolutionizing Data Connectivity: Introducing StreamNative's Universal Connectivity (UniConn) for Seamless Real-Time Data Access

In the new age of AI-driven applications, the ability to leverage data effectively hinges on the capability to stream and access it in real-time. The rapid movement of data necessitates robust connectivity to external applications and data stores, serving as both the source and destination of this data. However, enterprises often face significant connectivity challenges that can impede their ability to fully utilize their data.

Connectivity Challenges

Enterprises encounter a myriad of connectivity challenges that can hinder their data integration efforts:

1. Diverse Systems: Integrating a variety of systems and applications can be complex without standardized connectors, making seamless communication difficult.
2. High Development Costs: Building and maintaining custom connectors require significant resources and skilled developers, leading to high costs.
3. Security and Compliance Risks: Ensuring secure data transmission and compliance with regulations is challenging without robust, standardized connectors.
4. Scalability Issues: Custom connectors may not be optimized for performance, causing scalability issues as data volumes grow.
5. Operational Challenges: Monitoring, debugging, and managing data flows can be difficult with custom-built connectors, leading to increased downtime and operational inefficiencies.

Introducing UniConn: Unified Connectivity by StreamNative

StreamNative is excited to announce the launch of UniversalConnectivity (UniConn) in Public Preview for StreamNative Cloud. UniConn provides a consistent and declarative experience to connect, process, debug, and monitor data pipelines powered by Kafka Connect or Pulsar IO connectivity frameworks. This innovative framework facilitates seamless data movement in and out of StreamNative Pulsar or Kafka clusters, and allows users to perform lightweight data transformations using their preferred programming languages such as Java, Go, or Python.

• Seamless Pipeline Building: Build robust pipelines with Kafka and Pulsar connectors, ensuring flexibility in technology choice.
• Consistent User Experience: Enjoy a unified experience for developing, debugging, and monitoring pipelines with both Kafka and Pulsar.
• Flexible Connector Options: Use built-in connectors or bring your own, whether custom-built, open-source, or from partners.

Introducing Kafka Connect In StreamNative Cloud

StreamNative has long supported Pulsar IO, a framework for building connectivity with Apache Pulsar. With the introduction of UniConn, StreamNative now extends support to run Kafka Connect-based connectors within StreamNative Cloud. Kafka Connect, an open-source framework, is designed for developing connectors that link external data stores to Kafka clusters.

• Scalable Data Integration: Efficiently integrates data between Kafka API-compatible systems and various systems.
• Pre-Built Connectors: Wide range of connectors from community, and ISVs.
• Distributed and Fault-Tolerant: High availability and automatic failure recovery.
• Simplified Data Movement: Abstracts data ingestion and export complexities.
• Easy to Manage: Built-in tools for simple deployment and management.

Users can now log in to StreamNative Cloud to access the newly added connectors in the Connector Catalog. These connectors are available under the Kafka Sinks and Kafka Source tabs. 

Through the StreamNative Cloud Console UI, users can create, debug, and monitor Kafka Connectors seamlessly.

Create Connectors In StreamNative Cloud

Users can quickly build a data pipeline by creating a connector in just seconds. StreamNative Cloud's built-in connector catalog provides a wide selection of connectors, including the newly added Kafka Connectors. Users simply select a connector, enter the required configuration, and deploy the connector with ease.

Debug Connectors In StreamNative Cloud

StreamNative Cloud offers robust debugging capabilities for connectors, ensuring users can troubleshoot issues efficiently. Users have full access to connector logs, which can be viewed directly within the Console UI or routed to a Kafka topic for integration with logging services such as Datadog, Elastic, and others. 

‍

Monitoring Connectors In StreamNative Cloud

Once a connector is operational and data is flowing between the external system and Kafka, users can monitor its performance by viewing connector metrics in the Connector Dashboard or exporting the metrics to observability platforms such as Prometheus or Grafana. StreamNative Cloud offers a comprehensive set of metrics, enabling users to monitor various aspects of connector performance. Learn more about the Kafka Connect metrics supported by StreamNative Cloud here.

‍

Support for Single Message Transforms (SMTs) in StreamNative Cloud

Kafka Connect Single Message Transforms (SMTs) are lightweight transformations applied to individual messages as they pass through Kafka Connect. They allow users to modify, filter, or manipulate messages without writing custom code. SMTs are commonly used to alter message structures, add or remove fields, or apply simple logic such as routing, masking, or format conversions. These transformations help streamline the integration process between data sources and sinks, enhancing data flow between Kafka and external systems.

The newly launched Kafka Connect functionality in StreamNative Cloud fully supports Single Message Transforms (SMTs), allowing users to apply real-time transformations to individual messages within their data pipelines. This feature enables seamless data manipulation and customization without the need for additional coding. 

New Kafka Connectors in StreamNative's Built-In Catalog

‍

StreamNative is thrilled to announce the initial launch of four connectors within its built-in catalog, now available with Kafka Connect support in StreamNative Cloud. These connectors include:

• Milvus Sink
• MongoDB Sink
• MongoDB Source
• Iceberg (OSS)
• Yugabyte CDC Source

These new connectors enable seamless integration and data flow between Kafka and these popular data stores and services, enhancing the overall connectivity experience for users. Overtime StreamNative plans to add more connectors to the UniConn catalog based on the demand.

Scaling Connectors without High Costs

UniConn offers a scalable solution for running connectors on demand, without the worry of increased costs associated with higher levels of parallelism. This allows users to scale connectors efficiently and cost-effectively.

User-Provided Connectors

UniConn supports the ability to bring your own connectors to StreamNative Cloud, whether they are built in-house, by an open-source community, or a third-party vendor. Users can upload and self-manage these connectors while

Connector Portfolio In StreamNative Hub

StreamNative Hub provides more than 50 connectors in StreamNative Hub. You can filter connectors by Kafka Connect or Pulsar IO.

‍

Connector Shared Responsibility Model

StreamNative Cloud operates under a shared responsibility model for connectors. Enterprise support for fully managed connectors is provided by StreamNative, while support for user-provided connectors involves a shared responsibility between StreamNative and the user. This model ensures robust support while empowering users with flexibility and control over their connectors.

‍

‍

Custom Connectors: Customers are responsible for self-managing these connectors. StreamNative does not provide support for custom connectors or any other open-source connectors uploaded by the customers to StreamNative Cloud. 

Partner Connectors: These are connectors which are built and supported by StreamNative partners.

Fully Managed Connectors: These are connectors which are built and supported by StreamNative as fully managed connectors in StreamNative Cloud.

Conclusion

With the launch of UniConn, StreamNative is set to revolutionize data connectivity, offering enterprises a robust, scalable, and cost-effective solution to meet their real-time data integration needs. Explore the possibilities with UniConn and transform your data connectivity experience today.

Visit StreamNative’s website to learn more about UniConn and explore Kafka Connect documentation. Transform the way you connect, process, and manage your data with the cutting-edge capabilities of StreamNative Cloud and UniConn.

‍