How Lakestream turns a stream into lakehouse tables, ClickHouse tables or search indexes, with no connector in between.
Yesterday, we open-sourced Ursa, UFK and the Lakestream API and specification, and said that reading a stream is only half of what the specification covers. The other half is materialization, where a stream becomes a table in the catalog you already govern. This post is the closer look we promised.
Streaming data usually exists twice. It's written once as a stream, and then something copies it into the shape people need to query: a table for analysts, a column store for dashboards, a search index for whoever is on call. That copy is a pipeline, and every pipeline is one more thing to keep in sync with its source.
The Stream Materialization Framework makes the queryable shape a declared property of the stream itself. You declare the shape a stream should take, and the framework builds and maintains it.
What is the stream materialization framework?
The framework is the part of Lakestream that turns a stream into another queryable shape: a lakehouse table in Iceberg or Delta, a ClickHouse table, an OpenSearch index, and more as new materializers are written. Streaming data lands as governed assets in the catalogs you already use.
It's declarative. A policy on a namespace or on a single stream says what shape the data should take and where it should live. The framework maps the stream schema to the destination schema, evolves it as the source schema changes, retries failed writes, routes records that cannot be written to a dead-letter path, and commits the results. The declaration is part of the Lakestream API and specification, and Ursa implements it.
Each stream is materialized during compaction into the shape it's declared as.
Where it came from: compaction
Ursa writes a stream to object storage through a topic-mixed write-ahead log: one log shared by many topics. Small objects make an append durable quickly and cheaply, but they're slow and expensive to read back later, because a single read has to touch many of them. That read amplification is why a compactor rewrites them in the background into larger, columnar, indexed objects, which keeps reads fast and is also how storage gets reclaimed.
To do that, compaction reads every record once, decodes it with its schema, and writes it out in a new format. With all of that already happening, writing the same records into an Iceberg table is one more output from the same pass. Ursa's lakehouse integration began there, and StreamNative Cloud's one-click Kafka-to-Iceberg experience is built on it.
Then people asked for ClickHouse, and then for search indexes. The lakehouse path had been built into compaction directly, so each new destination meant building the whole stack again: schema mapping, schema evolution, retries, failure handling. We pulled that machinery out into a framework instead, with one pluggable materializer for each destination.
Why it matters
Traditionally, this work is handled by connectors or stream-processing pipelines. At a high level, they all perform the same transformation: read records from a stream, map them into a destination schema and representation, and write them into another system. In practice, that means reading a Kafka topic and writing an Iceberg table, a ClickHouse table, a MongoDB collection, or an OpenSearch index.
Each destination integration has to address many of the same concerns: schema conversion, schema evolution, retries, failure handling and dead-letter routing. Each connector or processing pipeline also introduces another runtime to deploy, scale, monitor and operate. The cost multiplies with every new connector and every new destination. The source and the destination usually aren't governed in the same catalog, either, so they're treated as two pieces of data.
Compaction already has every record and its schema in hand. That makes it the natural place to derive the destination's schema and write the destination's format too: once, in one framework, for every shape. The stream and what it becomes are governed in one catalog. At StreamNative, beyond Iceberg, we've supported ClickHouse tables, MongoDB and OpenSearch indexes on the framework.
Because compaction works directly against object storage, materialization adds no load to the brokers serving producers and consumers. It requires no consumer groups and no separate connector cluster, and its compute scales independently from the brokers. The one-click Kafka-to-Iceberg experience becomes a generic one-click stream-to-table experience.
Where it's going
Once the shape is a declaration, the shape can be anything a materializer knows how to build, including storage built for a single workload.
Take agent traces. An agent's work is a tree of calls that branch and retry. The framework could compact trace data into a tree-aware structure built for querying those trees, while ingestion stays on the Lakestream API, or on the Kafka protocol through UFK (Ursa for Apache Kafka), at full throughput. Anyone who can write a materializer can build that kind of structure, optimized independently for how the data will be queried.
We want Lakestream to be an open ecosystem, and materialization is one of the places that matters most: new shapes, built by people with workloads we haven't seen. The API and specification are still evolving.
Contributions welcome
The contributions that would help most:
- A materializer for a shape we don't have yet.
- A materializer written in another language, such as Rust.
- Feedback on the declaration in the specification, especially from anyone who has built a pipeline it should replace.
To get started:
- Specification and documentation: openlakestream.org
- Code: github.com/openlakestream/ursa
Apache, Apache Iceberg, Iceberg, Apache Kafka and Kafka are trademarks of the Apache Software Foundation. Other names and logos are trademarks of their respective owners.
