CyberGym

Why it's ranked here

The scoring rule is the reason to take it seriously. A crash alone does not count: the input has to separate the broken build from the patched one, which filters out lucky fuzzing. Results must use a single designated final answer rather than best of many, a rule the maintainers added after seeing that crediting any successful attempt rewards brute force as models improve. Submissions must also report tokens, time and cost per task. The catch, in the maintainers' own words: the strongest models with unlimited resources have nearly saturated it, so raw success rates now separate systems less than cost does.

What's good

The anti-cheating work is concrete. The documented setup swaps real task identifiers for random 12-character codes, so an agent cannot look up the original bug report by name. An optional firewall puts agent containers on a Docker network with no route out and forces traffic through a Squid proxy that only permits listed domains, so a program that ignores proxy settings still cannot escape. Four difficulty levels set how much the agent is told, from source code alone up to the patch itself. The FAQ names the specific leaks to strip before handing over a runnable container: the git history and the reference crash input.

Tradeoffs

It is heavy. The task data is about 240GB, the full set of Docker images is about 10TB, and even the static binary-only mode needs about 130GB; a ten-task subset exists for trying it out. The scoring server ships with a public placeholder API key, and its main submission endpoint, which executes uploaded inputs against real unpatched software, needs only a per-task checksum rather than that key, so the README insists it never face the public internet. Python 3.12 or newer is required. Results sit in a single local SQLite file, and the repository tree shows no test directory.

How to use it well

Use it to measure an offensive-security agent or compare models on reproducing known bugs, on a host you control. Start with the ten-task subset, which mixes five tasks agents solve and five they mostly do not. Bind the server to the Docker network gateway rather than every interface, and turn on the firewall if the agent gets network access, then read the transcripts anyway: the FAQ warns that web search routed through model APIs can still get past it. If you publish numbers, report the final-submission rate and the cost. It scores reproduction of already-fixed bugs; it is not a scanner for your own code.

Observed

License

Apache 2.0

Language

Python, requires 3.12 or newer

Runtime requirement

Docker, for task builds, PoC execution and the optional egress proxy

Install surface

Editable pip install with dev and server extras; benchmark data cloned from Hugging Face via git LFS

Interfaces

Python CLI modules for task generation, the scoring server and the firewall; HTTP API for PoC submission and verification

Task sources

ARVO and OSS-Fuzz

Difficulty levels

Four, from vulnerable source only up to patched source plus patch diff

Success criterion

PoC crashes the vulnerable build and not the patched build; final designated submission is the required metric

Storage footprint

About 240GB task data, about 10TB full server images, about 130GB binary-only mode, ten-task subset available

Network isolation

Optional Squid domain-allowlist proxy on an internal Docker network

Tests

No test directory in the repository tree

Publication

ICLR 2026 conference paper