← All IntelClip / AI AgentsEvaluating an LLM router as an n-by-n classifier
From Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber · ≈7:43
Reusable eval technique: treat routing as classification and measure precision/recall per destination branch, which generalizes to multi-model cost/latency routing.
What’s in it
- Reusable eval technique: treat routing as classification and measure precision/recall per destination branch, which generalizes to multi-model cost/latency routing.
Clip transcript
last step is we want to decide whether or not we should enhance or skip. How do we actually eval this? This is you could think of this as a more sort of traditional classifier. So, here we we have a confusion matrix. You know, many of you are probably pretty familiar with this. Um but we can look at things like the true positive cases, the false negative negative cases, and so on and so forth. Essentially, what we're doing is we're measuring the precision recall. In practice, your routers might actually be much more sophisticated. So, for example, we might want to route an image to a lower latency smaller model to be able to save on cost and improve the user experience at the trade-off of quality. And if that's the case, instead of having a 2 by 2 matrix for your confusion matrix, you might actually have an n by n matrix. Where each grid is actually telling you whether or not you're correctly routing to that specific branch.
Comments
Checking sign-in…
Loading comments…