olmo-eval: an open harness for scoring every checkpoint

olmo-eval is Ai2's open evaluation workbench for the repeated testing that happens while a language model is being built. Every change to training data, architecture or settings produces a new checkpoint, a saved version of the model partway through development, and each one has to go back through the same benchmarks to show whether it improved. Ai2's announcement, which Intel also carries on its own, presents the tool as an extension of OLMES, the standard Ai2 released in 2024 to fix the prompt formatting and task wording that had varied between papers and made scores hard to reproduce. olmo-eval carries that consistency from published comparisons into day-to-day development.

The design separates what is measured from how it is run. A task defines the questions and scoring, a suite groups tasks, and a harness controls how a task runs, so the same task can be scored once as a plain baseline and again with tools or other scaffolding without changing what it measures. The model, tools, container environment and helper models such as an AI grader can each be swapped. Benchmarks that require the model to use tools in an isolated sandbox get that heavier setup, while the rest use a lightweight default. Every run, its configuration and its results are recorded in one structured format, so checkpoints can be grouped and compared over time.

The comparison features address what an overall average hides. A results viewer lines up two checkpoints question by question, so a small but real change becomes visible, and each score comes with a standard error and a minimum detectable effect, the smallest difference that can be reliably told apart from noise. That answers the question Ai2 uses as its example: whether a 2.4-point change is enough to act on. The post explains the design with code examples but reports no benchmark results, and the code is on GitHub.