Introducing Ori Eval: the easiest way to write your first eval.
There's no definitive best model, only the best model for each task. Ori Eval leverages OpenRouter's APIs for each task in your codebase, and then evaluates the results.
curl -fsSL https://t.co/ABRt1wxtZ4
Directly in your favorite coding agent, Ori Eval explores your codebase, finds where you call models, asks what’s important (cost, perf, etc), writes the eval, and tests every model candidate.
You get a ranked comparison, scored on your criteria.
Because Ori Eval is an agent, it pins the harness, the model, and the effort for the duration of a run.
It also comes pre-tuned, so you don’t have to.
An eval checks three things: the tool it called, the tool it avoided, and the quality of the answer.
run.tool(‘search’).toBeCalled()
run.tool(‘delete_file’).toNotBeCalled()
run.toComplete()
Open-ended answers get graded by LLM-as-a-judge.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
Why it matters
It turns model choice into a test you keep: describe a bug in plain words, get an evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → that fails on it, then leave that assertion in CI so the regression cannot reach production.
Read the full thread on X ↗
