Names the benchmarks that now measure end-to-end exploit capability and shows a large capability jump in Mythos Preview, useful for judging model risk and what defenders must assume.
Key takeaways · AI-distilled
Anthropic says no public exploit benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → was hard enough to measure Claude Mythos Preview at release, so it ran it on two new academic benchmarks, ExploitBench and ExploitGym, plus an updated SCONE-bench; it beat all other evaluated models on all three.
ExploitBench, from CMU and Bugcrowd, scores complete exploits rather than proofs of concept, using 41 patched V8 bugs and 16 programmatically verified capabilities in five tiers, from reaching the vulnerable code up to arbitrary code execution.
Scoring needs no human or LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judge: lower tiers use differential execution against the patched build, higher tiers replay challenge-response checks across randomized heap layouts so hardcoded leaked addresses fail, and a transcript scan flags other cheating.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
