olmo-eval is built for the checkpoint you score fifty times
- Source
- x.com
- Date
Building an LLM means evaluating it over & over as it changes. Tweak a hyperparameter or scale the model up, & every new checkpoint sends you back through the same benchmarking loop. We're releasing olmo-eval, a workbench built for this kind of iterative model development. 🧵
olmo-eval builds on our OLMES project, which made benchmark scores comparable & reproducible by standardizing how models are evaluated. But a final score is only part of the story—olmo-eval works across the intermediate experiments teams compare throughout model development.
Running every benchmark in a locked-down sandbox – as many eval platforms do – is compute-heavy. So olmo-eval instead treats benchmarks differently depending on their runtime needs. For example, a plain Q&A benchmark runs directly—faster & cheaper than sandboxing.
In olmo-eval, every component is swappable: the model being evaluated, its tools, LLM-as-a-judge graders, & more. You can change one without touching the rest. Benchmark results land in a uniform schema, so checkpoints stay comparable across a long project.
Context
Ai2's olmo- helps developers see what changed between saved versions of a model. It records evaluation settings and results in a common format, then lines up two versions' answers to the same questions while holding other conditions fixed. That makes improvements and regressions visible at the question level, where an overall average can conceal them. The workbench description also separates the from how the model runs, allowing developers to change a component without rebuilding the entire test.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
Checking sign-in…
Loading comments…