Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber
Source
AI Engineer
Author
AI Engineer
Date
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
Why it matters
multimodalA model that works with more than text — reading images, audio, or video, and sometimes generating them too.Full definition →evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → design with no established playbook is one of the hardest open problems in shipping agents.
Key quotes
“You want to start with your logging cuz if you don't start with it, you have nothing to optimize for, let alone set up a self-learning loop.”
Jai Chopra
“For routing, our guardrail metric is recall. We don't want any bad image to slip through our system.”
Soumya Gupta
“there's a chance your model's going to hallucinate these two extra wings to to match the description”
Soumya Gupta
“You will have one static model with the offline, but this is what is going to keep your system alive.”
Soumya Gupta
“we think of this like a Swiss cheese model. So, we want to try and optimize for reducing the chance of a failure getting into production.”
videoBuilding uReview, Uber’s Multi-Agent Code Review Engine — Will Bond & Ameya Ketkar, UberAI Engineer
videoAI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDashAI Engineer
videoHow Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube AdsAI Engineer