ExploitGym
cybergym.io- Category
- Cybersecurity
- Rank
- No. 585Tools index
- Listed in
- Find AI benchmarks
- Pricing
- Free
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Interfaces
- Web
- Date
About
Evaluates whether agents can turn a known vulnerability and a triggering input into a working exploit in controlled evaluation environments.
What it does
A benchmark suite of 869 tasks across userspace, browser engine, and kernel code, where an AI agent is given source code, a crash-inducing input, and a sandboxed environment, then must retrieve a hidden flag to demonstrate real code execution.
Stated on the product site
- Task Structure
- Each task supplies the agent with source code, a proof-of-vulnerability input that triggers the bug, and a containerized runtime to work in.
- Scope
- The task set covers three domains: userspace programs, the V8 browser engine, and the Linux kernel.
- Evaluation Method
- Standard mitigations such as ASLR, stack canaries, and a heap sandbox can be enabled or disabled per run, and a separate judging step confirms an exploit actually used the provided vulnerability.
- Reported Metrics
- Results include per-task averages for cost, output tokens, time, and number of LLM calls.
- Access Model
- Evaluation is run under trusted access arrangements intended for security research, with notes accompanying each submission.
Not stated on the site
- The page does not state whether the task set, evaluation harness, or containerized environments are available for others to use, or under what terms.
- It is not specified how or whether outside teams can submit their own agents to be evaluated on this benchmark.
Written from the product site at cybergym.io.
What it can do
Provide benchmark tasks that test AI agents' ability to convert a vulnerability and proof-of-vulnerability input into a working exploit
Known vulnerability and proof-of-vulnerability input → Working exploit achieving code execution
Evaluate exploits for bypassing mitigations like ASLR and sandboxes
AI-generated exploit code → Success/failure assessment against mitigations
Tags
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.