Intel / repo

lm-evaluation-harness v0.4.12

Source: github.com
Author: baberabb
Date: Published May 11, 2026

Context

The v0.4.12 release of lm-evaluation-harness, software for testing language models, fixes errors that could change what a actually asks a model. One fix stops preprocessing from corrupting answer text containing brackets in GPQA. Another restores question text dropped from RACE prompts.

These are corrections to the measuring instrument. A score can change because the model receives the intended question after a fix, even when the model itself is unchanged. For comparisons spanning evaluator versions, the documented fixes therefore give a concrete reason to check the affected task setup. The release does not quantify how much earlier scores were affected.

Key takeaways · AI-distilled

MMLU-Pro's few-shot examples were leaking into the user role once a chat template was applied, so any chat-template score from before this release was measuring a malformed prompt rather than the benchmark.
The GPQA preprocessing regex was stripping bracketed math out of answer text, quietly corrupting the strings being graded. Eval bugs like this surface as plausible-looking numbers, not crashes.
Asking for thinking mode on multiple-choice tasks used to fail silently. The now rejects enable_thinking for loglikelihood-scored tasks outright and requires a think_end_ whenever thinking is on.
Duplicate task or group configs inside the same root are now skipped with a log line instead of silently overwriting each other, so two YAML files claiming the same task name become visible instead of random.

Terms in this piece · Glossary

agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Key quotes

“New release with four new model backends, tensor parallel support for `transformers` based models (`hf`), new benchmarks, a `TaskManager` refactor, and a long tail of task correctness fixes.”

“Duplicate task/group configs within the same root are skipped with a log message instead of silently overwritten.”

Read the source github.com