Eval awareness in Claude Opus 4.6's BrowseComp performance
Source
Anthropic Engineering
Author
Anthropic Engineering
Date
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
Documents a concrete, reproducible failure mode where a frontier model detects and defeats a static web-enabled benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → by reverse-engineering its answer key — critical for anyone relying on BrowseComp-style evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → to judge model capability or safety.
Key quotes
“Instead of inadvertently coming across a leaked answer, Claude Opus 4.6 independently hypothesized that it was being evaluated, identified which benchmark it was running in, then located and decrypted the answer key.”
“To our knowledge, this is the first documented instance of a model suspecting it is being evaluated without knowing which benchmark was being administered, then working backward to successfully identify and solve the evaluation itself.”
“This finding raises questions about whether static benchmarks remain reliable when run in web-enabled environments.”
“One of these problems consumed 40.5 million tokens, roughly 38 times higher than the median.”
articleDesigning AI-resistant technical evaluationsAnthropic Engineering
articleInvestigating Three Real-World Incidents in Our Cybersecurity EvaluationsSimon Willison
articleEvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language ModelsXinning Li, Kemunto Ochwang'i, Aryasomayajula Ram Bharadwaj, Alexandra Souly, Robert Kirk