Designing evals and hillclimbing without fooling yourself

Anthropic has added two commands to the claude-api skill that ships inside Claude Code. The first, /claude-api build-eval, builds an evaluation inside your codebase: a set of test tasks plus a grader that scores the outputs. The second, /claude-api hillclimb, then improves the application against that evaluation one change at a time. Hillclimbing means making a small change and keeping it only if the score goes up. ClaudeDevs announced the release with a guide by Lance Martin. Most of the guide is about how that loop can fool you: the grader can be wrong, the score can move by chance, or a change can fit the test cases without helping real use.

build-eval starts with the inputs. It draws on production transcripts first, after asking about retention and sensitive data. Then come bug reports and tickets, five to ten handwritten cases, and finally cases synthesized from the codebase. It shows every input on a page and waits for you to confirm them. Next, it proposes the cheapest grader that fits. If the possible outputs are constrained, that is a code check such as an exact match or a schema check. If the output is open-ended, a second model acts as the judge, scoring against a rubric written as checkable claims, and that judge should not be the model under test. Claude then grades a handful of cases and asks whether you would have scored any differently. During the baseline runs it runs the grader twice on the same output to see whether the verdict changes. It also checks for timeouts, API errors and cut-off answers. If the baseline already scores about 95% or higher, it warns that there is little room left to improve quality and that the climb should target cost or latency instead.

hillclimb asks what you want to optimize, such as performance, or cost while performance holds. It then splits the cases at random into a train set it may read and a test set it never sees. Before the first round, it checks that chance variation in the score is smaller than the smallest improvement you would act on, and suggests more repetitions or cases if it is not. Each round, Claude reads the previous train transcripts and proposes one change as a patch. If the train score rises but the test score stays flat, it suspects overfitting and reverts. It also reverts on a regression, and keeps the patch only when both sets improve. Overfitting here means tuning to the particular evaluation examples, for instance adding an OCR tool that helps the benchmark but rarely helps production. When the score stalls for two or three rounds, Claude sorts the remaining failures by cause, which can expose a flawed task, a harness error or plain variance. At the end, it leaves the code at the version that did best on the test set and reports the result against the baseline with confidence intervals. If the gain is within noise, it recommends against merging.

The article's two examples are Anthropic's own internal evaluations. The first is a 44-ticket customer support benchmark, with 30 tickets used for the search and 14 held out. The starting setup was Opus 4.8 at high effort, scoring 74.4% on the search tickets at 4.6 cents per ticket. The final configuration was Sonnet 5 on low effort with prompt fixes. It reached 98.9% on the search tickets, but the number that tests generalization is the held-out one: 90.5% against the original setup's 78.6%, at about one fifth of the cost. The second example tuned the claude-api skill itself against a documentation-derived evaluation, which rose from 66% to about 88%. Part of that work fixed the evaluation rather than the skill: one task's grader demanded more than the task asked for, and another grader's instructions contradicted the docs.