PostTrainBench

Why it's ranked here

It measures something few evaluations touch: whether an assistant can run a real research loop end to end, with no human in it, and produce a better model. The scoring is careful rather than naive. Flagged or broken runs fall back to the untuned model's zero-shot score instead of keeping a gamed number, contamination is decided by a majority of three judge runs, and the headline metric weights each task so easy gains do not dominate. MIT licensed, with a published paper behind it.

What's good

The sandbox is strict in a checkable way. Each agent starts with a cleaned environment and receives only the provider keys its own allowlist plus the task's grading needs, and every other provider key stays out of the sandbox. The agent gets the same overlap checker and test-set copy the contamination judge uses, so it can self-police. A hard timeout adds only five minutes past the budget. The full task prompt, including the rules against training on test data or swapping in an instruction-tuned model, is printed in the README.

Tradeoffs

This is research infrastructure, not a weekend install. The reference pipeline assumes an HTCondor cluster with Apptainer containers and an overlay filesystem, and the README contradicts itself on cloud support: one note says a Harbor adapter runs everything on Modal, another says Harbor support is planned. Judges run through a Codex subscription login, so a lapsed ChatGPT session stops the grading phase. The Claude agent runs with permission checks disabled. The README names six target tasks, so check which set any published score used.

How to use it well

Read it as a scoring methodology first and a leaderboard second. If you compare agents on it, compare the hour budget, GPU count, agent variant and model alongside the number, because each result folder name encodes all four. Use the reruns over several seeds before trusting a small gap. Running it yourself makes sense for a lab with cluster or Modal access and several provider accounts; it does not tell you how an assistant handles everyday coding.

Observed

License

MIT

Language

Python and Bash

Supported agent harnesses

Claude Code, Codex CLI, Gemini CLI, OpenCode (plus Cursor and Grok CLI auth handling in the job script)

Reference scheduler

HTCondor with Apptainer containers; cloud path via a Harbor adapter on Modal

Sandbox isolation

Agent container launched with a cleaned environment and a per-agent API key allowlist

Score fallback

Contaminated or errored cells fall back to hardcoded zero-shot baseline scores

Evaluation framework

inspect_evals installed from a pinned commit in the container