Session Overview
Discover how Harvey built an in-house data agent to automate ad-hoc analytics and scale internal data access.
In this talk, Connor and Jingyi (Data Scientists at Harvey) break down how they designed a portable context layer to teach their internal AI agent, Spectra, company-specific metrics, business rules, and table quirks. Learn why pointing LLMs directly at a data warehouse fails, how to structure modular skill files in dbt, and how to set up robust offline evaluation, model selection, and team rollouts to handle 1,000+ weekly questions with confidence.
What You'll Learn
- Why frontier models struggle with internal company context and database nuances
- Structuring an agent context layer: Analytics triage harness, key data assets, and foundational vs. leaf skills
- Anatomy of a skill file: Descriptions, trigger words, definitions, gotchas, and minimal examples
- Creating an offline evaluation benchmark: Sub-agents, expected outputs, and hill-climbing accuracy
- Side-by-side model evaluation: Why the largest frontier model isn't always the best fit for internal questions
- The transition from spending 20% of team time on ad-hoc triage to maintaining code-reviewed skill repos
- Step-by-step guidance on implementing your own modular context layer
Chapters
- 00:00 — Introduction & the ad-hoc data triage problem
- 02:23 — Why pointing agents straight at Snowflake fails
- 03:32 — Writing company context into dbt skill files
- 04:15 — Architecture map: Harness, data spine & modular skills
- 07:11 — Step-by-step request flow: Entity resolution to SQL planning
- 10:00 — Anatomy of a skill file: Trigger words, definitions & gotchas
- 13:09 — Offline evaluation: Benchmarks & multi-agent scoring
- 15:46 — Model evaluation & selecting the right default LLM
- 18:04 — Beta testing, cross-team rollout & security boundaries
- 20:21 — Impact at Harvey: 1,000+ weekly queries answered
- 21:40 — How to get started on your own data agent
