CryptanalysisBench tests whether language models can find attacks on cryptographic designs. Its 191 tasks distinguish schemes with known practical weaknesses, schemes without known practical breaks tested at full and reduced strength, and a separate challenge set of production cryptography.
The authors report that the five tested models broke between six and twelve full-strength schemes in the second tier. They also report previously unknown findings involving two designs. Keeping the tiers separate is essential: success against these benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → targets does not establish that deployed cryptography in general has been broken.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleAnthropic Analyzes Mythos 5's Unsanctioned Attacks in AISI Cyber Testing
articleClaude Formally Verifies 123 Open-Source Libraries at the Binary Level
articleAnthropic's Scion: Source Control Rebuilt for Agent-Scale Merges
articleAnthropic Details How It Cut Claude Distillation by 90 to 99 Percent
articleDeepsecBenchMalte Ubl
articleAre the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon StatementsXinke Tong, Xuanming Zhang, Tianyi Tang, An Yang, Jiatu Hu, Guojie Lin, Zhenzhen Shi, Lingfeng Zeng, Boyu Yang, Bing Zhao, Hu Wei, Lin Qu, Dayiheng Liu
articleLlm Novice UpliftScale AI blog