TL;DR

StreamNative addresses the challenge of cost-effective, scalable data streaming to lakehouses by introducing its Ursa engine. This solution leverages object storage to minimize inter-zone traffic and integrates seamlessly with catalogs like Amazon S3 Tables, Snowflake Open Catalog, and Databricks Unity Catalog. The result is a reduction in costs and complexity, enabling real-time applications to access streaming data as tables efficiently.

Opening

In the rapidly evolving landscape of AI and data-driven applications, organizations face the dual challenge of scaling their data infrastructure while managing costs. Traditional streaming solutions often incur hidden expenses related to inter-zone network traffic and complex data pipelines. StreamNative's Ursa engine offers a breakthrough by providing a cost-effective way to stream data directly to popular lakehouse formats like Iceberg and Delta Lake, seamlessly integrating with leading catalog services to ensure data discoverability and governance.

What You'll Learn (Key Takeaways)

• Efficient Data Streaming: StreamNative's Ursa engine reduces inter-zone traffic, enhancing cost-efficiency by utilizing a leaderless architecture and direct object storage.
• Seamless Catalog Integration: Built-in support for catalogs such as Amazon S3 Tables, Snowflake Open Catalog, and Databricks Unity Catalog simplifies data management by automatically registering streaming data as queryable tables.
• Real-time Data Availability: Ursa enables real-time applications to access and query streaming data in table formats, facilitating immediate insights and decision-making.
• Scalable Architecture: The engine is 100% compatible with the Kafka API, supporting scalable and flexible data streaming solutions tailored to modern cloud environments.

Q&A Highlights

Q: Who manages the table lifecycle, Ursa or the catalog provider?
A: The lifecycle management depends on whether the tables are external or internal. For external tables, management is by the catalog provider; for internal, it's by StreamNative, ensuring a single data copy that backs streaming and historical reads.

Q: Can you use your own compaction tools?
A: Custom compaction tools may not be supported for internal tables, but StreamNative collaborates with customers to optimize data maintenance and compaction strategies.

Q: How is compatibility with schema evolution managed?
A: Currently, schema evolution is not supported but is on the roadmap. StreamNative is actively working on enhancing this feature based on customer feedback and evolving needs.

Q: What permissions are needed for setup?
A: Permissions are required for catalog integration, object storage writing by StreamNative, and data reading by catalog providers like Databricks or Snowflake. Comprehensive setup guides are available through StreamNative Academy.