Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

An AI-edited food photo can look better while misrepresenting the meal. Uber's engineering team describes a system for Uber Eats that checks both visual quality and faithfulness to the original. A routing step first decides whether an image needs editing. If it does, an editing agent receives feedback from a separate quality checker and can try again within a fixed attempt limit. Edits that still fail are withheld from publication. The talk describes this as a production workflow.

Human judgments also shape the automated checks. The team samples production images for human labeling, compares those labels with the agents' decisions, and tests revised configurations against a reference dataset before deployment. Automation handles the revision loop; human-labeled examples continue to define acceptable behavior.

“You want to start with your logging cuz if you don't start with it, you have nothing to optimize for, let alone set up a self-learning loop.”

Jai Chopra

“For routing, our guardrail metric is recall. We don't want any bad image to slip through our system.”

Soumya Gupta

“there's a chance your model's going to hallucinate these two extra wings to to match the description”

Soumya Gupta

“You will have one static model with the offline, but this is what is going to keep your system alive.”

Soumya Gupta

“we think of this like a Swiss cheese model. So, we want to try and optimize for reducing the chance of a failure getting into production.”

Jai Chopra