DeepsecBench: Vercel's Unseen-Codebase Cyber Leaderboard
- Source
- vercel.com
- Author
- Guillermo Rauch
- Date

Context
OpenAI's new GPT-6.2 models hold the top four places on Vercel's DeepsecBench, a leaderboard that measures how well AI models find security vulnerabilities in application code. Guillermo Rauch of Vercel posted the results on the day OpenAI released the models, calling them "the most cybersecurity-capable models in the world." The leaderboard supports a narrower statement: on this test, GPT-6.2 Astra at its highest reasoning setting (xhigh) scored 61.22, followed by Astra at medium (53.81) and two GPT-6.2 Sol runs (47.35 and 44.74). The best entry from another lab, Claude Sonnet 5.5 at its max setting, scored 37.50 in eighth place.
The score rewards finding real bugs more than avoiding false alarms. According to Vercel's launch post in July, the uses one open-source codebase frozen at a commit just before many vulnerabilities were fixed. Vercel picked 50 entry-point files and built a reference set of 231 findings judged by humans. Recall is the share of those 231 a model finds, and precision is the share of everything it reports that turns out to be real, with any finding outside the reference set ruled real or false by a judge model. The score combines the two while weighting recall twice as heavily, because missed vulnerabilities go unfixed while false positives do not make code less secure, and each published result is the median of three runs. GPT-6.2 Astra xhigh found 56.2% of the known findings with 95.0% precision, for $59.85 and just under an hour of run time. When Vercel launched the benchmark, its best run found 30.7%.
Rauch's description of DeepsecBench as a "stealth" benchmark refers to how Vercel keeps it secret. The launch post says Vercel does not disclose the repository, the commit, the files or the findings, so there is nothing for models to train against, and argues that a model reciting memorized fixes would score near-total recall. Those same design choices limit what the ranking means. It is one codebase and 50 files, a judge model decides which extra findings count, and the leaderboard's column shows models running in different tools (Codex for OpenAI, Claude Code for Anthropic, Pi for the rest). Anthropic's Fable 5 does not appear because, Vercel said in July, it declines security work, including defensive tasks. The result shows GPT-6.2 finding far more of one hidden codebase's known vulnerabilities than any other model tested, which is strong evidence on that task rather than a full ranking of cyber capability.
- GPT-6.2 Astra at xhigh effort tops DeepsecBench at 61.22, with 56.2% recall and 95% precision for $59.85 and 56 minutes per run. At medium effort it scores 53.81 for about half the cost, $30.37.
- OpenAI models running in Codex hold the top seven places. The best non-OpenAI entry is Claude Sonnet 5.5 at max effort in Claude Code, scoring 37.50 for $65.25 and taking over six hours.
- Claude Opus 5.5 at max effort scores 26.57 with 15 false positives, below Claude Opus 5 at max (32.31), which was also the most expensive configuration listed at $127.93.
- Precision and recall diverge sharply: Grok 4.6 at xhigh reports zero false positives but only 13.3% recall, and DeepSeek V4.1 Flash at high effort scores 24.71 for $0.56, a fraction of the frontier entries' cost.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Checking sign-in…
Loading comments…



