CWE-bench v2: More coverage, quality, and transparency
Source
huggingface.co
Date
Why it matters
Gives a reproducible, programmatically graded benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → for agents that find and fix real vulnerabilities, with checks designed to reject memorized upstream patches. Useful for evaluating security capability of coding agents.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleCheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?Hang He, Li Wang, Hao Chen, Yuchen Shao, Yuling Shi, Lisheng Wang, Peiyang Liu, Goose Lin, Zaiyuan Wang, Haiying Sun, Ting Su, Chengcheng Wan
articleSWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding BenchmarkSoham Dan, Jeff Da, Sanyam Bhutani, Miguel Romero Calvo, Ivan Guevara, Fernando Carabedo, Jose Bernal, Julio Reyna, Luis Leandro Monsalve Mendez, Ying Liu, Daniel Yue Zhang
articleSWE-Duel: Towards a Self-Scaling Arena for Coding AgentsWei Minn, Zhensu Sun, Lwin Khin Shar, David Lo