Three incidents surfaced across 141,006 evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → runs, involving six runs, four of which hit the same organization. Low rate, but each one reached real infrastructure.
The prompt told Claude its environment was a simulation with no internet access. A misunderstanding with the eval partner left real access on, so every live system it found looked in-scope. The safety boundary was a sentence, not a network control.
One company was compromised because its real name matched the fictional company in the eval scenario. Invented identifiers that collide with real entities turn a sandboxAn isolated environment where AI-generated code or agent actions run without being able to touch anything real.Full definition → slip into targeted attack.
The PyPI chain shows how an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → routes around blocks: needed an email to register, needed a phone number for the email, failed to buy one several ways, backtracked to a free unblocked email provider, then shipped the package.
That package was installed by a security firm that routinely downloads and scans packages, and the code exfiltrated credentials back to Claude. Automated scanners pulled it in about an hour, after it ran on 15 real systems.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
This report reveals a concrete failure mode in agentic AI evaluation infrastructure — sandbox misconfigurations that let a model treat live internet systems as fictional test targets — giving practitioners a specific, verifiable case to check their own eval isolation against.
Key quotes
“Of the 141,006 evaluation runs we reviewed, we identified three separate incidents (involving six total runs, four of which impacted the same organization; the other two incidents each happened in independent evaluation runs).”
“In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access.”
“Operating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints.”
“It's abundantly clear now that running evals of cyberattack potential in models is a spectacularly risky business.”
