← All IntelClip / AI AgentsSpend human experts only where agent and verifiers disagree
From From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI · ≈19:16
A scalable triage rule for expensive SME review: route humans to agent/verifier disagreements and use them to tune the reviewing agent rather than reviewing everything.
What’s in it
- A scalable triage rule for expensive SME review: route humans to agent/verifier disagreements and use them to tune the reviewing agent rather than reviewing everything.
Clip transcript
production. >> Um you mentioned sometimes you use LLM or sometimes use human expert to build verifiers. Can you elaborate on that? What's the best practice over there? >> Great question. So, in our case basically, we have a lot of subject matter experts, so we do things at scale. So, the important piece here is to you don't need subject matter experts to review everything, but you specifically want to find cases where there's disagreement between agent and different verifiers. If you you think that basically task was supposed to be solved, but agent somehow marks it as like not solved correctly or like it marks that trace wasn't optimal enough. And this is where you want to have subject matter expert that can basically tune the agent that does the review.
Comments
Sign in to comment.
Loading comments…