Safety blocks hit defensive cyber work: CyberGym results and costs
- Source
- ArtificialAnlys
- Date
Restricting offensive without blocking defensive is difficult, as finding and proving a vulnerability requires the same steps whether the goal is to exploit it or patch it On CyberGym-E2E-AA, a benchmark that measures cyber defense capabilities from discovery to patching on memory-safety tasks, some frontier intelligence models are safety blocked from responding on 85%+ of tasks. The good news is that some of the most capable models are also the most cost effective. With GPT-6 Luna or MiMo-V2.6-Pro, you can run ~100 bug hunts in a 1M+ line codebase for ~$20 - up to 100x cheaper per task than the next most capable model Grok 4.7.

Some frontier models refuse over 85% of defensive vulnerability tasks because discovery steps look identical to offense. Pick the model with this in mind: GPT-6 Luna or MiMo-V2.6-Pro run about 100 bug hunts for roughly $20.
postSolar Mini 4: cheap tokens, expensive tasks, weak agentic coding
postArtificial Analysis launches Cyber Index for AI cyber defense evaluation
postGPT-6.1 Sol reaches near-Astra intelligence at a quarter of the cost per task
postSonnet 5.5 nears Opus 5.5 on agentic evals but burns record output tokens
Checking sign-in…
Loading comments…

