Session Overview
Learn how to reliably test and improve non-deterministic AI systems using automated evals and experimentation frameworks.
In this talk, Felipe Perez, a member of the technical staff at OpenAI, walks through how they built an internal enterprise agent platform and evaluation harness. Discover how to move beyond simple question-and-answer benchmarks to multi-turn evaluations, test against prompt injections and tool failures, and use A/B-style statistical testing to ship agent improvements without causing regressions.
What You'll Learn
- Agent anatomy: Tracking how prompts, policies, tools, and untrusted inputs impact agent context
- Multi-turn evaluations: Testing real production workflows with deterministic checks and LLM-as-a-judge graders
- In-code evals: Writing unit tests for tools and faking trajectories to catch prompt injection vulnerabilities
- Experimentation loops: Running statistical tests over diverse task sets to balance safety, latency, and cost trade-offs
