Ori Eval adds cost pilots and blocks same-family LLM judges
- Source
- x.com
- Date
We shipped Ori Eval last week, to help developers like you find the best model for what you're building. Since the launch, we received great feedback, suggestions, bug reports, and more, from the community. Here are five major improvements we shipped this week:
1/ You can now see why a model failed When a model comparison fails with a score of, say, 0.82 when the bar was 0.75, Ori Eval will now tell you what caused it to fail. The report also names the provider that served each model, so you know where the result came from.
A run that crashed will no longer be reported as a "wrong answer" Crashed runs are now reported as failures.
2/ You will now know how much an eval costs before it runs We added "ori eval --pilot" which runs every case once and reports an estimate (within 5%) of the full run.
Context
OpenRouter updated Ori to make model comparisons easier to interpret before acting on a ranking. Following the initial release, a pilot run can estimate the cost of the full evaluation, and crashed runs are distinguished from completed wrong answers. The update thread also says the tool warns when the model grading answers belongs to the same family as a candidate and asks for another judge. These changes help separate model quality from execution failures and make the grading arrangement visible.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Checking sign-in…
Loading comments…