Session Overview
Learn how Together AI designs event-driven pipelines to handle billions of interactions per day across open-source model inference workloads.
In this session, Abhinav (Staff Software Engineer at Together AI) discusses the tension between immutable event streams and real-time billing decisions. He breaks down the architectural shift from slow batch jobs to streaming infrastructure built on Apache Kafka, Apache Flink, and data lakes. Discover how decoupling raw usage facts from business logic makes billing, re-rating, and backfills straightforward, and see how Together AI uses an onboarding agent to codify and automate new data pipelines end to end.
What You'll Learn
- The core challenge of metering and pricing high-throughput GPU inference
- Why decoupling immutable usage facts from volatile pricing logic prevents costly reprocessing
- How Together AI structures its stream pipeline with Kafka durable queues, Flink validation gates, and Delta Lake
- How to enforce idempotency and at-least-once processing across pipeline segments
- Reusing the same streaming blueprint for telemetry, GPU fleet health, and CRM streams
- Using an AI onboarding agent to generate schemas, SDK bindings, and Flink jobs from a single spec
Chapters
- 00:00 — Introduction & the inference metering problem
- 02:04 — Moving from batch jobs to streaming architecture
- 04:01 — Decoupling immutable usage facts from pricing logic
- 05:32 — End-to-end streaming architecture: Kafka, Flink & Delta Lake
- 07:32 — Achieving idempotency & delivery guarantees across segments
- 09:51 — Reusing the event streaming spine across new use cases
- 10:47 — Automating pipeline scaffolding with an onboarding agent
- 13:50 — Key takeaways: Making billing & event pipelines boring
