MMLU-Pro's few-shot examples were leaking into the user role once a chat template was applied, so any chat-template score from before this release was measuring a malformed prompt rather than the benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →.
The GPQA preprocessing regex was stripping bracketed math out of answer text, quietly corrupting the strings being graded. Eval bugs like this surface as plausible-looking numbers, not crashes.
Asking for thinking mode on multiple-choice tasks used to fail silently. The agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → now rejects enable_thinking for loglikelihood-scored tasks outright and requires a think_end_token whenever thinking is on.
Duplicate task or group configs inside the same root are now skipped with a log line instead of silently overwriting each other, so two YAML files claiming the same task name become visible instead of random.
Terms in this piece · Glossary
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
If you evaluate open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → LLMs, this release adds first-class backends for TensorRT-LLM, Megatron-LM, Gaudi, and a LiteLLM gateway plus native multi-GPU tensor parallelism for HF models — letting you benchmark the same tasks across far more inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → stacks. Watch the breaking changes (vLLM >=0.18, SteeredHF renamed, stricter enable_thinking rules) before upgrading.
Key quotes
“New release with four new model backends, tensor parallel support for `transformers` based models (`hf`), new benchmarks, a `TaskManager` refactor, and a long tail of task correctness fixes.”
“Duplicate task/group configs within the same root are skipped with a log message instead of silently overwritten.”
