livenerf

Why it's ranked here

Most claims that a model got worse arrive with no baseline. This one was committed in public before data collection, uses the statistics of a published error-bar paper, and reports its own blind spots. Its validation run found that lower effort cut output tokens by 62% while the accuracy drop stayed inside its error bars, and that swapping in the previous Opus was not distinguishable at all; the project prints both results rather than hiding the second. MIT licensed and built on the UK AI Security Institute's Inspect evaluation framework. Narrow by design, one model from one machine, but rigorous about what it can and cannot see.

What's good

Each call runs with no tools, no plugin servers, a fixed empty working folder, and any environment setting that could switch model, effort or billing stripped out. A sample is discarded, never scored, when a different model answered part of it, when the turn was retried, or when the context grew past a size cap that signals something extra was loaded. Graders are plain functions with no model acting as judge. The daily runner refuses to start if the tool version or the frozen question list has changed, and it skips a run when the subscription usage meter is high, then retries hourly so a day is not lost.

Tradeoffs

The detection floor is coarse: about 7.5 accuracy points per ten-day window as designed, 6.6 measured after the baseline, so smaller drops read as no change. Every run comes from one account, one machine and one region at a fixed morning hour, so regional or peak-load serving differences go unseen. Each call carries the account email and the questions are public, so provider-side special treatment cannot be ruled out. The question audit, done by Claude itself, flagged 8 suspect answer keys and 30 ambiguous questions, a conflict of interest the project discloses. The decision rule needs two consecutive windows after the baseline, so no call is possible before day 30.

How to use it well

Treat it as an instrument, not a scoreboard. Anyone arguing that a Claude model got worse can check daily scores against the launch-week baseline, and should watch output token counts first: the project's own validation found reduced thinking shows up there before accuracy moves. Running a copy needs Python 3.11 or newer, the uv package manager and a logged-in Claude Code install, with spend set in points of your weekly plan meter. It does not cover the raw API, the chat app, long context, tool use or agent behavior, and a detected change says nothing about its cause.

Observed

License

MIT

Language

Python, 3.11 or newer

Install

git clone then uv sync; no package registry release described

Interface

Command-line modules run with python -m, plus an Inspect model provider for headless Claude Code

Core dependency

Inspect evaluation framework, pinned to an exact version

Platform support

Linux, macOS and Windows stated in the README; Windows scheduled-task installer included

Grading

Exact-match pure-function graders, no LLM judge

Prerequisite

A logged-in Claude Code install able to run claude -p