ExploitBench
github.com/exploitbench/exploitbench- Category
- Developer Tools
- Rank
- No. 1224Tools index
Previous survey · No. 1199 ·
- Listed in
- Find AI benchmarks
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- GitHub
- 444 stars
- Date
About
Measures how far agents can take real, hardened vulnerabilities toward a working exploit, scored on a ladder of verifiable capability tiers. Few exploits are public, which limits contamination but complicates reproduction.
What it does
ExploitBench is a Python harness that hands an AI model a container holding one real bug, gives it tools over the Model Context Protocol (MCP), and records how far up an exploitation ladder it climbs: reaching the vulnerable code, triggering the bug, building memory primitives, then arbitrary code execution. The first target set is Chromium's V8 JavaScript engine, scored on 16 capability flags. Each run is one model, one bug and one seed under a turn cap, and every transcript, tool call and grade is written both to a run folder on disk and to a local SQLite database.
Why it's ranked here
The grading is the strong part. Capabilities are flags a grader sets, not a judge's opinion, and they only accumulate, so a run's result is the highest rung it reached. Bug environments pass a five-step validator before use, including a check that a known proof of concept still reproduces, and an audit command can replay scored exploits in a fresh container to catch ones that hardcode memory addresses. MIT licensed, Python 3.11 or newer, Linux. The limits are scope and weight: one engine today, a package marked alpha, and per-bug images of roughly 70 GB.
What's good
Cost control is built in. A mock-model run costs nothing, a real-model test run is quoted at four to ten cents, and a spend cap stops scheduling new runs once total cost crosses it, recording the skipped ones so a retry flag brings them back. Image tags are resolved to fixed digests at run start, so a registry retag fails the run instead of silently swapping the target. Tools reserved for the grader are hidden from the model, and environments where those overlap the agent's tools are rejected at load time, closing an obvious way to game the score. Crashed or timed-out runs can resume by replaying their tool calls.
Tradeoffs
This is heavy infrastructure. The setup check warns below 100 GB of free disk and fails below 20 GB, Docker is required, and the declared platform is Linux. The documentation disagrees on matrix size: the main README describes the full config as 41 V8 bugs, while the benchmarks guide calls it 7 models across 14 bugs and 5 seeds, 490 episodes. Token budgets weight output five times input, which the code says is exact for Anthropic and approximate for OpenAI and Gemini. Results also shift with optional mid-run hint prompts, so two published numbers are comparable only if both runs used the same setting.
How to use it well
It fits security researchers and labs measuring how far a model gets on real browser-engine bugs, with API budget and a Linux machine that has Docker and disk to spare. Follow the quick start order: the free mock run, the setup check, then one bug, one seed, a 100-turn cap and a small spend limit before any sweep. Pin images by digest for anything you publish, and keep hint prompts off unless you are matching the older scaffolded baseline. Only V8 configs ship; other task types need environments you bring. The authors ask that it not be used for reinforcement learning training.
Technical notes+
exploitbench/runner/loop.py holds the single agent loop: it filters manifest-declared evaluation tools out of the MCP tool list before the model sees them, merges each grade result into a cumulative OR of capability flags (never demoting a flag), and stops early once the ace flag is set. exploitbench/runner/budget.py weights tokens as base input plus cache creation plus 0.1 of cache reads plus 5 times output (CACHE_READ_WEIGHT, OUTPUT_WEIGHT), sends a stuck nudge after 50 turns without a grade call and a one-time wrapup at 75% of the turn budget; docs/architecture.md still states the older formula without base input or the output weight. exploitbench/contract/manifest.py validates manifests against a Draft 2020-12 JSON Schema, then requires episode and evaluation tools to be disjoint and forces cli_oneshot grading when the interface demands it. exploitbench/validator/runner.py sequences five checks (manifest schema, MCP contract, target starts, known PoV reproduces, integrity posture) and rolls them up to fail, unvalidated, pass_unverified or pass_verified. exploitbench/runner/orchestrator.py bounds concurrency with one asyncio.Semaphore, dedupes on a UNIQUE(benchmark_id, model, env_id, seed) insert for resume, and wraps each tuple in an episode timeout (default 1800 s). pyproject.toml points pytest at exploitbench/tests, which was not among the files read.
Observed
- License
- MIT
- Language
- Python, requires 3.11 or newer
- Platform
- POSIX Linux (package classifier); Docker required
- Interface
- Typer command-line tool installed as exploitbench, plus a FastAPI read API command
- Agent tool surface
- MCP server over stdio inside a Docker container per bug
- Model providers
- Anthropic native SDK; OpenAI, Gemini and OpenRouter via LiteLLM; any OpenAI-compatible gateway
- Storage
- Local SQLite database mirrored by per-run directories on disk
- Development status
- Alpha (package classifier)
- Usage request
- README asks that the benchmark not be used for reinforcement learning
Read from README.md, pyproject.toml, LICENSE, docs/architecture.md, exploitbench/cli.py, exploitbench/runner/loop.py, exploitbench/runner/orchestrator.py, exploitbench/runner/budget.py, exploitbench/validator/runner.py, exploitbench/contract/manifest.py, benchmarks/README.md.
Intel on ExploitBench
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.