Session Overview
Discover how to build, test, and govern an autonomous SRE agent to triage and resolve Apache Kafka production incidents.
In this talk, Jeremy (Field CTO at Lenses.io) walks through designing an AI incident management system that wakes up before your on-call engineer does. Learn how to orchestrate supervisor and specialist sub-agents using Amazon Strands and Model Context Protocol (MCP), safely inspect live Kafka infrastructure without risking outages, run reproducible failure injection harnesses, and enforce fine-grained policy guardrails.
What You'll Learn
- The on-call reality: Why Kafka consumer lag and stalled consumers cost businesses thousands during war rooms
- Multi-agent supervisor pattern: Coordinating triage, Kafka diagnostics, code forensics, and report generation
- Live triage via MCP: Connecting secure MCP tools to PagerDuty, GitHub, Confluence, and Lenses
- Testing & evaluation harness: Safely provisioning clusters, generating shadow traffic, and injecting poison pill commits
- Observability & cost tracking: Tracing agent trajectory, tool usage, and LLM token costs with Arize Phoenix
- Safe execution with Dogwood policies: Catching and validating agent tool calls before they execute
- Fast vs. slow thinking: Leveraging decision models to drop latency, reduce costs, and evaluate root cause quality
Chapters
- 00:00 — Introduction & the Kafka on-call dilemma
- 04:02 — Why testing SRE agents requires breaking infrastructure
- 06:00 — Architecture overview: Supervisor agent & MCP tools
- 08:22 — Live demo: Triaging consumer lag & poison pills
- 13:29 — Pop the hood: Amazon Strands multi-agent design
- 15:45 — Building an incident evaluation & replay harness
- 19:06 — Tracing agent trajectory & token costs with Phoenix
- 21:30 — Safe agent governance: Memory, prompt injection & Dogwood policies
- 25:17 — Faster, cheaper evaluation with decision models
- 28:44 — Future vision: Auto-drafting PRs, dashboard checks & CI guardrails
- 31:06 — Cost comparison: Claude Sonnet vs. Haiku & summary
