The Lakestream API and specification, the Ursa storage engine, and Ursa for Apache Kafka are open source today under Apache 2.0, at openlakestream.org.
We are excited to announce that we are open sourcing Ursa, the stream storage engine that powers the managed Pulsar and Kafka services on StreamNative Cloud, and UFK (Ursa for Apache Kafka), the Kafka distribution that uses Ursa as its diskless storage. The design behind it won the best industry paper award at VLDB 2025. Here are the most important pillars of the Lakestream vision:
- Open source API and specification. An API and specification aimed at providing a standard for building data streaming storage on top of object storage, and at enforcing interoperability between different data streaming protocols at the storage layer. It's still evolving, and we'd love the community to shape it with us.
- Open source implementation. We're open sourcing Ursa, our implementation of the Lakestream API and specification.
- Stream materialization framework. The API and implementation for materializing a stream into lakehouse tables, extensible to materialize streams into other queryable states.
- Open source diskless Kafka. We're open sourcing our Kafka distribution, which adds diskless topics to Apache Kafka: a topic can keep its records on object storage through Ursa instead of on broker disks.
- Multi-protocol support. With its open Lakestream API and storage library, Ursa can serve as the storage backend for different streaming systems, including but not limited to Kafka and Pulsar. The open build ships with Kafka today; Pulsar is coming.
Why open source?
In the context of supporting real-time applications and AI agents with a Streamhouse, we need an open stream storage foundation that supports diverse streaming and messaging protocols, not just for today but for the innovations of the coming decades.
Unfortunately, most messaging & streaming systems today are broker-based, which means the storage is hidden behind a protocol and isn't exchangeable. A protocol tells you how to talk to a system. It says nothing about the bytes after they land, or whether anything else can read them without going through the broker first. This leads to duplicated effort, inefficiency and network overhead.
Diskless is the right trend to address these challenges. However, diskless systems, ours included until today, still hide the implementation behind a specific system and a specific protocol ecosystem. That results in siloed data, wasted resources and fragmented governance across data assets. And without a solid interface built over open data lakes, the same effort is duplicated for every system.
Tables had this problem once, and open table formats solved it. So we're taking their lesson and providing an open storage foundation for streams.
Streams need what tables got
Parquet standardized the file. Iceberg and Delta standardized the table on top of it. Tables stopped belonging to whoever wrote them, and any engine could read the same copy without negotiating a single integration.
Streams never got that, and for most of their history they couldn't. Durability came from replication between brokers, so the records, the consensus protocol and the process that ran them were one artifact. There was no layer to open. Object storage breaks that apart: it keeps the bytes durable on its own, which leaves the streaming system holding a layout and a contract. Both can be written down.
We aren't the only ones who noticed. The Apache Kafka community is moving onto object storage too, in KIP-1150, and we expect to benefit from that work. But every KIP reaches exactly as far as Kafka. Its committers decide what it says and when it changes, and that is how it should work. A format is different: any system can implement it, and no one has to ask.
So we wrote the format down. The Lakestream API and specification say how a stream is laid out in a bucket, how an append becomes durable, how offsets are assigned, how a cursor reads across what's been written and what's been compacted, and how a closed segment is registered as a table. Publish those contracts and any system that implements them can read the stream, without depending on the process that wrote it.
Reading a stream is half of it. The specification also covers materialization, so a stream can become a table in the catalog you already govern. When it does, the operational and analytical halves of a company's data stop being two copies of the same records, drifting on different schedules.
What the specification deliberately leaves out is the protocol. Lakestream standardizes the storage layer underneath streaming systems, and the semantics your clients see belong to the engine above it. We know that split holds because we run it both ways in production: the Pulsar service and the Kafka service in StreamNative Cloud sit on the same Ursa storage underneath.
Two protocols on one storage layer is the small version of a bigger picture. A lakehouse and a Streamhouse aren't rival buildings. They stand on the same foundation: object storage, open formats, and the catalogs that govern both. Open table formats opened that foundation for tables. Lakestream is our attempt at the same thing for streams: an open storage layer anyone can implement.
The category belongs to the whole industry, though. Ververica coined the Streamhouse name, and last week we joined four other companies in the Streamhouse Working Group, which published an open definition built on three attributes: real-time, production-native and decentralized. That definition is vendor-neutral, and it doesn't require Ursa or any other particular technology underneath.
Ursa 1.0 Release
That open storage layer starts with Ursa 1.0, which includes:
- The Lakestream API and specification: the stream catalog, the log, cursors, the data format, and the integration that makes a stream readable as a table. We expect it to change as others implement it.
- The storage engine: the Java implementation of the Lakestream API and specification, built to be embedded as a library in messaging brokers. Implementations in other languages are coming soon.
- The stream materialization framework: a framework and implementation for materializing streams into lakehouse tables, so streaming data lands as governed assets in your data catalogs. It also materializes streams into other queryable states, including ClickHouse tables and OpenSearch indexes, and gives you one abstraction for bridging streams with the rest of your governed data. We will publish a separate blog post that goes inside it.
UFK Release
UFK shows what embedding Ursa looks like. It's our Kafka distribution: a fork of Apache Kafka that adds Ursa as diskless storage. Here's what's built into it:
- Diskless topics. A topic created with Ursa storage enabled writes its records to object storage through Ursa instead of to broker disks. Durability comes from the object store rather than from replication between brokers.
- Everything else stays Kafka. Every other topic keeps its local storage and replication, in the same cluster, under the same controller quorum, ACLs and group coordinator. The wire protocol is unchanged, so existing clients and tools connect as they do today.
- Topics as governed assets. A diskless topic can also be materialized into a lakehouse table or another queryable state, through the stream materialization framework.
UFK differs from other diskless designs in where the diskless path lives: inside Kafka's replica manager, where UFK routes a diskless topic's storage operations to Ursa. It lets us add a diskless path without changing the system around it, and it lets both kinds of topics share one cluster. The same design makes it possible to move a topic between disk-based and diskless storage, which this release doesn't include yet. Upstream Kafka has no pluggable interface for the replica manager, so UFK is a fork today; we'd love to work with the community to contribute some work back so we don't need to maintain a separate fork.
Ursa doesn't know that Kafka exists. UFK is one way to use it, and we want it to be one of many.
An open ecosystem around stream storage
We'd love to see what people build on Lakestream. An engine for another protocol, sitting on the same storage the way UFK does. An implementation of the API in another language. A materializer for a system we haven't implemented yet. We expect other people to build services on the same specification, including ones that compete with ours.
That's the ecosystem we want to build around stream storage: open stream storage as a solid addition to open table formats, for both the lakehouse and the Streamhouse. We'd love the community to work together with us to shape the API and specification.
We didn't start the move to object-store-native data systems. It's been going on for a few years now, in streaming, in vector databases, in search, in a growing number of distributed systems projects that all found the same thing: object storage changes what a data system has to be. We're adding a layer underneath that everyone can implement.
What's next
Our own next steps begin with Pulsar. Ursa started as a storage system for both Apache Kafka and Apache Pulsar, and Ursa storage will be open sourced to Pulsar in future Pulsar releases.
We've also built integrations that use the Lakestream API directly, bypassing brokers: one for Kafka Connect and one for Kafka MirrorMaker. They'll be open sourced soon too. Universal Linking, our commercial product, is built on the same idea.
The goal is an open streaming ecosystem on object storage, and a good foundation for real-time data. Contributions of every kind are welcome, to the specification as much as to the code.
Get started today
The specification is public, the implementation is public, the license is Apache 2.0, and your data lands in your bucket in open formats, reachable through open interfaces.
Read the specification. Point a coding agent at it. Build something exciting and fun.
- Specification and documentation: openlakestream.org
- Ursa code: github.com/openlakestream/ursa
- UFK code: github.com/openlakestream/kafka
Apache, Apache Kafka, Kafka, Apache Pulsar and Pulsar are trademarks of the Apache Software Foundation. These projects are not affiliated with or endorsed by the ASF.
