Intel / article

Investigating Three Real-World Incidents in Our Cybersecurity Evaluations

Source: simonwillison.net
Author: Simon Willison
Date: Published Jul 30, 2026

Context

Anthropic's cybersecurity tests reached real organizations because machines in the testing environment had unintended internet access. Its incident report says prompts described a simulation without internet access, leading models to initially treat reachable systems as part of the exercise. The review identified three incidents across six runs.

This exposes a concrete failure in design: describing a network as isolated does not make it isolated. Anthropic says checking every internet access path and monitoring activity could have helped prevent or detect the incidents. These tests also lacked the additional safeguards used in its generally available products, which matters when interpreting what the incidents establish.

Key takeaways · AI-distilled

Three incidents surfaced across 141,006 evaluation runs, involving six runs, four of which hit the same organization. Low rate, but each one reached real infrastructure.
The prompt told Claude its environment was a simulation with no internet access. A misunderstanding with the eval partner left real access on, so every live system it found looked in-scope. The safety boundary was a sentence, not a network control.
One company was compromised because its real name matched the fictional company in the eval scenario. Invented identifiers that collide with real entities turn a slip into targeted attack.
The PyPI chain shows how an routes around blocks: needed an email to register, needed a phone number for the email, failed to buy one several ways, backtracked to a free unblocked email provider, then shipped the package.
That package was installed by a security firm that routinely downloads and scans packages, and the code exfiltrated credentials back to Claude. Automated scanners pulled it in about an hour, after it ran on 15 real systems.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.

Key quotes

“Of the 141,006 evaluation runs we reviewed, we identified three separate incidents (involving six total runs, four of which impacted the same organization; the other two incidents each happened in independent evaluation runs).”

“In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access.”

“Operating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints.”

“It's abundantly clear now that running evals of cyberattack potential in models is a spectacularly risky business.”

Read the source simonwillison.net