Session Overview
How do you answer complex data questions across 70,000 tables, 600 petabytes of data, and hundreds of thousands of daily queries?
In this talk, Bonnie Xu (Data Productivity Lead at OpenAI) explains how OpenAI designed, built, and deployed an internal AI data analyst agent. Discover why traditional table metadata falls short, how data lineage and codebase context empower accurate SQL generation, and how automated evals and memory flywheels keep the agent reliable at scale.
What You'll Learn
- The challenge of table discovery, complex queries, and semantic drift at enterprise scale
- How OpenAI grounds its AI agent using metadata, data lineage, codebase tasks, and company institutional knowledge
- Performance and safety: Enforcing performant SQL generation, user permissions, and PII safeguards
- Testing and reliability: Using automated evals and reviewer agents to prevent regressions
- Core lessons learned: Tool minimization, prompt flexibility, and memory flywheels
Chapters
- 00:00 — Introduction & the challenge of enterprise data scale
- 01:23 — Why OpenAI needed an internal data agent
- 02:35 — Obstacles: Table discovery, nuances, and complex SQL
- 04:13 — Introducing the internal AI data analyst agent
- 05:07 — Live demo: Investigating New York City ride metrics
- 07:48 — Real-world investigation: Diagnosing a ChatGPT active user spike
- 11:25 — Agent architecture: Entry points, context, and backends
- 12:33 — Context layers: Table metadata and semantic relationships
- 14:24 — Context layers: Data lineage & codebase enrichment
- 16:49 — Context layers: Institutional knowledge & Slack integration
- 17:46 — Building a memory flywheel: Continuous improvement & shortcuts
- 20:02 — Enforcing query performance & SQL guardrails
- 22:36 — Verification, transparency, and confidence scores
- 23:42 — Evals & preventing model regressions
- 26:20 — Security, access control, and PII protection
- 27:40 — Lessons learned: Reducing tool sprawl & prompt flexibility
- 29:42 — How to get started & final takeaways
