Today we're open sourcing Streaming Proof, the harness we run against our own builds to answer one question about a streaming system: did it deliver every message it accepted, in per-key order, while something underneath it was failing? It's available under Apache 2.0 at github.com/openlakestream/streaming-proof.
What is Streaming Proof?
Streaming Proof is a correctness harness for streaming systems. It drives keyed, sequenced load through a Kafka-compatible system, Apache Pulsar or an MQTT broker, and while your chaos tooling kills pods and splits networks, it checks the guarantees the system says it keeps. Every message should arrive at least once, and messages with the same key should arrive in order. For Kafka transactions, each should arrive exactly once. During a run and at the end of it, the harness reports every missed, duplicated or out-of-order message, down to the key and the sequence number.
It isn't a benchmark. It generates enough traffic to expose a bug.
Why we built it
Measuring throughput takes a load generator and a clock. Correctness is harder. A broker that silently drops one record in ten million during a leader election doesn't show up as a latency spike or an error rate. It shows up much later, as a row missing from a reconciliation nobody expected to fail.
We build stream storage and run streaming services on it, so we need evidence that each build keeps its guarantees when brokers die and networks split. That need grows as durability moves from replicated broker disks to object storage. The failure modes move with it, and guarantees that held before have to be shown to hold again.
We looked for a tool that did this before we wrote one. Good tools exist, and each does part of the job.
- Jepsen set the standard for testing distributed systems under failure, and its analyses of Kafka-compatible systems are some of the most rigorous work in the field. It's built for deep investigations: it installs the system itself on a handful of nodes, typically runs for minutes, and analyzes the history afterwards. We needed something that runs every day, for hours, against the clusters we actually operate.
- OpenChaos, from the OpenMessaging project, is the closest in spirit. It checks several messaging systems for lost and duplicated messages, but it runs short, fixed tests, analyzes them afterwards, and has no Pulsar or MQTT driver.
- Apache Kafka's own test tools, its verifiable producer and consumer and its transaction tests, check for lost messages and exactly-once delivery. They're built for Kafka's system tests and speak only Kafka.
- The OpenMessaging Benchmark drives Kafka, Pulsar and MQTT, but it measures speed. Its consumers can't tell a lost message from a delivered one.
- Chaos tools such as Chaos Mesh are very good at breaking things. They don't check what arrived afterwards.
- Deterministic simulation finds bugs nothing else will, but the system has to run inside the simulator, and you still bring the checks yourself.
Long-running monitors exist for a single system, but each checks only part of the picture. None of these gave us the combination we needed. We wanted one harness that speaks several protocols and runs continuously against a real cluster at realistic scale. It had to check per-key ordering and exactly-once as well as delivery, run alongside the chaos tooling we already use, deploy like any other service, and tell us exactly which key and which sequence went wrong. So in early 2025 we built it.
How it works
To answer its one question, the harness has to know exactly what it sent and what arrived.
The coordinator compares what the producers sent with what the consumers received, while failures run underneath.
Every message carries a key and a sequence number that increases for that key, and each key has a single producer. Consumers keep what they receive as ranges per key. Receiving 1 through 5 and then 10 through 15 leaves two ranges and a gap from 6 to 9. Receiving 1 through 10 and then 5 through 15 leaves an overlap.
A coordinator collects checkpoints from the producers and consumers on an interval and compares them. A gap is a missed message, an overlap is a duplicate, and a sequence number that goes backwards within a key is an out-of-order delivery. At any single instant, a message that has been published but not yet consumed looks identical to a message that has been lost. Only elapsed time, and a rule about how much of it is too much, separates the two, so every unconfirmed message gets a timeout before it counts as missed.
The coordinator tracks, for each key, the highest sequence it has fully verified, and workers drop everything below that point, so memory stays bounded. And ranges are tagged with the moment a consumer opened them, so when partitions move between consumers during a rebalance, a message delivered twice shows up as an overlap instead of disappearing in a merge. Pulsar's Shared subscriptions deliver round-robin, so there the harness checks delivery and counts duplicates, and doesn't expect per-key order.
The checking logic is index arithmetic of the kind that passes its unit tests and still hides a bug, and a bug there means a wrong answer about somebody's production system. So the harness's own algorithms are specified in TLA+ and model-checked within small bounds whenever the checking code changes: range merging, gap detection, duplicate counting, watermarks, and the exchange between the coordinator and its workers. The specifications don't cover the Java implementation or the systems under test.
How to use it
- Build and deploy it. It needs Java 21, and there's no published container image yet, so build the image locally. Then bring it up with Docker Compose, or install it on Kubernetes with the Helm chart in the repository.
- Point a driver at your cluster. The drivers for Kafka-compatible systems, Pulsar and MQTT pass your client settings through unchanged, security configuration included.
- Declare what to check and how hard to push. Choose the guarantees (at-least-once, ordering, exactly-once) and the shape of the load: keys, partitions, producers, consumers and message rate.
- Break something. Streaming Proof doesn't inject failures itself. The repository includes Chaos Mesh templates for pod kills, pod failures, network delay and network partitions, all disabled until you turn them on.
- Read the report. It shows what was verified, missed, duplicated, out of order or timed out, and how long verification has stalled. For runs you leave unattended, set a stall limit, so that a run that stops making progress fails instead of quietly passing.
The repository's quick start walks through the whole loop against a local Kafka cluster.
Our vision: build the ecosystem
"Exactly-once" and "no data loss" appear in plenty of documentation, and most of it asks you to take the claim on trust. We'd like those claims to come with evidence anyone can reproduce.
Earlier this week we open sourced Ursa and the Lakestream API and specification, and invited other implementations. Every one of them, ours included, should be able to show that it keeps its guarantees under failure. We're open sourcing Streaming Proof so they can all be checked the same way, with the same harness.
Contributions welcome
The contributions that would help most:
- A driver for a system we don't cover. The driver interface is small, and the Kafka, Pulsar and MQTT drivers are the references.
- A counterexample: a scenario, or a model-checker trace, that breaks an invariant we believe holds.
- New failure scenarios, and new guarantees to check.
To get started:
- Code and issues: github.com/openlakestream/streaming-proof
- Design document and specifications: in the repository's doc and tlaplus directories
Apache, Apache Kafka, Kafka, Apache Pulsar and Pulsar are trademarks of the Apache Software Foundation. Other names and logos are trademarks of their respective owners.
