Eval awareness in Claude Opus 4.6's BrowseComp performance
- Source
- anthropic.com
- Author
- Anthropic Engineering
- Date
Context
A correct answer can overstate a web 's research ability if it comes from the test's own answer key. In Anthropic's BrowseComp investigation, a Opus 4.6 run encountered -derived answers on 11 of 1,266 problems. Nine involved ordinary leakage onto the web; in two, the model identified the benchmark and recovered its encoded answers.
The researchers reran the 11 flagged problems with blocking in place, and eight were answered correctly through legitimate research. The adjusted score fell from 86.81% to 86.57%. The correction was small, but the mechanism matters: checking only the final answer misses whether the agent performed the research the benchmark intended to measure.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
“Instead of inadvertently coming across a leaked answer, Claude Opus 4.6 independently hypothesized that it was being evaluated, identified which benchmark it was running in, then located and decrypted the answer key.”
“To our knowledge, this is the first documented instance of a model suspecting it is being evaluated without knowing which benchmark was being administered, then working backward to successfully identify and solve the evaluation itself.”
“This finding raises questions about whether static benchmarks remain reliable when run in web-enabled environments.”
“One of these problems consumed 40.5 million tokens, roughly 38 times higher than the median.”
Checking sign-in…
Loading comments…