DeepsecBench
- Source
- vercel.com
- Author
- Malte Ubl
- Date
Context
A high ranking on Vercel's DeepsecBench still leaves most known security flaws undiscovered. The asks models to inspect an application codebase containing 231 human-judged findings across 50 entry-point files. Vercel's methodology reports that the best run finds 30.7% of those known findings, and that 20 of 25 runs find under 20%.
The headline score is a recall-weighted F2, which counts finding known flaws twice as heavily as avoiding false reports. It is not a percentage of code proven secure. These results help compare scanning configurations, but the low coverage of known issues means a scan with few findings cannot establish that an application is safe.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
“Hacks are initiated from the outside, so the single best defense is finding vulnerabilities from the inside before attackers do.”
“Today, higher price does not buy proportionally more. Kimi K3 , from Moonshot AI, ranks eighth at a score of 17.56 for $12.38 on the high setting, half the top score for about a fifth of the cost.”
Checking sign-in…
Loading comments…