Artificial Analysis launches Cyber Index for AI cyber defense evaluation

Artificial Analysis's new Cyber Index scores AI agents on defensive security work, and its first results show that a model's score depends on whether it will do the work as well as on whether it can. In its launch post, Artificial Analysis said five frontier models declined tasks worth 32 to 38% of the index on safety grounds and finished 19 to 31 points behind the leaders, despite what it called frontier agentic coding capabilities.

The index combines three benchmarks, each testing a different part of finding and fixing vulnerabilities, and launched alongside an alliance whose launch partners are Collinear AI, IBM, NVIDIA and Vercel. CWE-Bench-AA, from Collinear AI, covers auditing and patching in 120 held-out tasks that span all ten categories of the 2025 OWASP Top 10, a widely used list of web application security risks, across C/C++, Go, Java, JavaScript/TypeScript, Python and Rust. Vercel's DeepsecBench-AA tests discovery alone: given a codebase and a budget, the agent must find every vulnerability, and its findings are scored against a set produced by human security reviewers, with real findings rewarded and harmless code flagged as vulnerable penalized. Vercel's original write-up of that benchmark is covered in DeepsecBench. CyberGym-E2E-AA, from Berkeley RDI, runs the whole sequence on memory-safety bugs, errors where a program reads or writes memory it should not and can crash as a result. The agent must find the bug, write a proof of concept that triggers the crash, and patch the code so the crash no longer reproduces.

At launch, Grok 4.7 (xhigh) and MiMo-V2.6-Pro tied for first at 56, followed by GPT-6 Luna (max) at 53, GLM-5.3-Flash at 50 and Muse Spark 1.3 (xhigh) at 44. The models held back by refusals were GPT-6 Sol, GPT-6 Astra, Claude Opus 5.5, Claude Fable 5.1 and Gemini 3.8 Flash, each at the setting Artificial Analysis listed. Most of their gap comes from CyberGym-E2E-AA, where OpenAI's GPT-6 Sol and GPT-6 Astra refused every task and Anthropic's Claude Opus 5.5 and Claude Fable 5.1 refused 98% and 99%. It is the only one of the three benchmarks that asks the agent to write code that deliberately triggers a crash, though the post does not say which step the models declined.

A low Cyber Index score can therefore mean that a model refused the work rather than failed at it. For a security team comparing models, that distinction decides the result: under the settings tested, a model that declines the end-to-end task cannot be used for it, however strong its coding. Artificial Analysis returned to these refusals in a later post on CyberGym results, and a later index update added trusted-access models to the rankings.