SEC-Bench Pro
sec-bench.github.io- Category
- Cybersecurity
- Rank
- No. 2224Tools index
Previous survey · No. 2109 ·
- Listed in
- Find AI benchmarks
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Interfaces
- Web
- Date
About
Evaluates long-horizon security bug hunting in complex systems, including browser engines and the Linux kernel, with reproducible validation.
What it does
A benchmark site that ranks AI coding agents by how often they can find and prove exploitable bugs in real codebases such as browser engines and the Linux kernel, running each attempt inside a timed container harness and scoring outcomes automatically.
Stated on the product site
- Scope
- The benchmark aggregates results across three target codebases, V8, Firefox, and the Linux kernel, into a single overall leaderboard.
- Test harness
- Each benchmark task runs inside a Docker container with a rendered prompt and a time limit, and the agent's output files are collected for automated checking afterward.
- Task design
- An earlier version of the benchmark's proof-of-concept task offers three modes that vary how much information is given to the agent, from repository code alone up to a full sanitizer crash report.
- Task categories
- Tasks are split into two categories, one for producing proof-of-concept exploits for known vulnerabilities and one for generating patches that fix them.
Not stated on the site
- Whether the benchmark's task instances and harness are available for the public to download or run independently is not stated on the page.
- The page does not specify cost or compute resources required to run an agent through the harness.
- How new target projects or tasks get selected for future additions to the benchmark is not detailed beyond a note that more will be added over time.
Written from the product site at sec-bench.github.io.
What it can do
Evaluate AI agents on long-horizon security vulnerability-hunting tasks
AI agent → Task success results
Generate proof-of-concept exploits for real-world software targets
Target codebase (e.g., V8, Firefox, Linux) → Proof-of-concept exploit
Tags
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.