ProgramBench

Why it's ranked here

The interesting part is not the task idea, it is how hard the harness works to avoid flattering a submission. Build time network access is cut inside the container, unattempted instances count as zero rather than being dropped from the average, and a submission's own numbers can be recomputed without Docker or re-run end to end with it. Unreliable tests are excluded by recorded reason, including tests that pass on a stub binary. The evaluation module carries a written warning that a silent skip is worse than a loud failure. That is a scoring philosophy, and it shows.

What's good

Design choices are documented where they bite. The container layer explains why a timed out command needs an in container process sweep, and why archives are streamed through the container's own archive tool instead of the plain copy command, so symlinks and modes survive. Network blocking states its own limits: raw IP connections still work and a root process could undo it. Submission URLs are restricted to http and https to avoid local file reads. Scores are checked against a hash of the built artifact, and heavy log data can be split out and pulled back losslessly.

Tradeoffs

The images are built for linux on x86 only, so a Mac or Windows host is out unless you accept emulation. Docker is required for anything past the cheapest verification tier, ten CPU cores per container is the default, and test archives are pulled from HuggingFace on demand. The task set is fixed at whatever ships in the package, so adding your own program is not a supported path. Test suites were themselves model generated, which is why a list of exclusion reasons exists at all. Some evaluation internals are private helpers, so building on them means reading source.

How to use it well

This is for people measuring coding agents, not for people shipping features. It fits a research or model evaluation loop: run your agent offline against the containers, collect one archive per task, then hand the run directory to the harness and read the printed summary rather than raw per test output. Third parties auditing someone else's claimed numbers get the most value, since the cheap tier needs only the Python package. It does not host or run your agent, and it does not measure patch style tasks on existing repositories. Bring your own agent framework.

Observed

License

MIT, copyright Meta Platforms, per LICENSE

Language and floor

Python, requires 3.10 or newer per pyproject.toml

Install surface

Published Python package, installable with pip or uv, one console entry point

Interface

Command line only: evaluate a run directory, print a summary, sync test blobs, submit

Platform

Container images are built for linux on x86 only; macOS and Windows hosts need emulation

External dependencies

Docker for evaluation, HuggingFace for on-demand test archives

Data layout

One directory per benchmark instance with a task manifest and a test list, shipped inside the package

Verification

Two tiers: recompute scores from submitted output without Docker, or re-run the full evaluation

Build isolation

DNS blackholed inside the build container; raw IP traffic and root override are documented as not covered

Tests

A pytest suite is present in the tree covering XML parsing and score filtering