Session Overview
Discover how Apache Kafka serves as the event-driven backbone and shared memory layer for multi-agent AI systems.
In this talk, Stephane Derosiaux, CTO of Conduktor, explains how to architect agent swarms without running into memory drift, inconsistent state, or database bottlenecks.
What You'll Learn
- The 3 memory tiers: Private agent working state, Kafka Streams shared context, and vector stores
- Preventing state drift: Using Kafka as the single source of truth across specialized agents
- Solving slow LLM lag: How Kafka share groups handle queue-based consumption and message leases during long inference calls
- MCP integration: Exposing state stores directly to LLMs with Model Context Protocol
- Offline evaluation: Replaying immutable Kafka event logs to benchmark new models and agent behavior safely
Chapters
- 00:00 — Introduction & agent swarms on Kafka
- 05:02 — Single source of truth & turning the database inside out
- 07:18 — The 3 tiers of swarm memory
- 11:28 — Orchestration & decision models
- 13:23 — Solving LLM latency with Kafka share groups
- 18:13 — Interactive queries & MCP integration
- 24:36 — Global context & backtesting via log replay
- 28:20 — Conclusion
