← All IntelClip / AI AgentsPrompt optimizer as reflect plus synthesize sub-agents with an agent store
From Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber · ≈12:09
Implementable auto-tuning structure: one sub-agent reflects on mismatches to find systemic issues, another rewrites the agent config, and versioned registration plus rollback keeps the loop safe.
What’s in it
- Implementable auto-tuning structure: one sub-agent reflects on mismatches to find systemic issues, another rewrites the agent config, and versioned registration plus rollback keeps the loop safe.
Clip transcript
Um so Jay is going to spend more time on the diagnosis side of it. What I want to do is zoom into the auto-tuning bit. And again, we're looking at routing, but this is how we tune every agent in our system. Uh so we start with a target agent, and we've already got these uh unseen eval samples from our humans. We go find out the mismatch and matches and call a prompt optimizer agent. Now, this itself is two sub-agents. There's the reflect agent and the up synthesize agent. What reflect does is it it just looks at the mismatches, tries to find remove any noise, find any systemic issues that might be in your data set, and reflect on it and send that feedback to the synthesize agent. Now, the synthesize agent takes this feedback. It has your agent config. It goes and updates your agent with the new config based on the feedback it's getting. And goes and benchmarks again. If this benchmark is passed, you actually register this new agent in the new agent store. And next time your production runs, you pick up the new version of the agent. And this is a closed-loop system as I mentioned, no human in the loop. We definitely have observability on the guardrails, quick rollback built in in case of any issues with the system itself.
Comments
Checking sign-in…
Loading comments…