ZeroBench
zerobench.github.io- Category
- Developer Tools
- Rank
- No. 1382Tools index
Previous survey · No. 1334 ·
- Listed in
- Find AI benchmarks
- Pricing
- Free
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Interfaces
- Web
- GitHub
- 72 stars
- Date
About
Challenging visual reasoning questions with distinct main-question and subquestion scores. Compare pass@1, pass@5, and tool access separately.
What it does
ZeroBench is a test set for AI models that read images: 100 hand-built puzzles, each pairing a picture with a question that demands several steps of visual reasoning, plus 334 smaller subquestions that break those puzzles into checkable pieces. You download the questions and images from Hugging Face, send them to your model with a fixed instruction to put the final answer in curly braces, and mark it right only on an exact match. Each question is asked five times, which gives three numbers: average single-try accuracy, solved at least once, and solved every time.
Why it's ranked here
A hard benchmark with honest bookkeeping. It was built so that, at release, no evaluated model got a single main question right on one greedy try, and the README now openly reports that 70% of questions have been solved at least once. The authors version their changes in public (three question revisions, a red-teamed grading fix that corrected 0.71% of gradings), embed a canary string so labs can filter it from training data, and the paper was accepted at ICML 2026. The repository itself holds no code files, only a README, a license and a results chart, so its value is the question set, not software.
What's good
The solved-every-time metric is the useful idea here: it separates a model that gets lucky once in five tries from one that reliably solves the problem, and the README reports it next to the other two. Results distinguish official runs from externally reported ones and mark tool-assisted runs separately, so a number with tools is not passed off as a number without. Older question versions stay downloadable on their own branch, so past results remain reproducible. The README also reports cost per question for each top model, which matters when a benchmark needs five samples per item.
Tradeoffs
With 100 main questions, every question moves a score by a full percentage point, so small gaps between models are within the noise of one or two puzzles. The repository holds no evaluation harness: the only code is a short scoring snippet in the README, and it leaves the model call as a placeholder you write yourself. Grading is strict string matching on the last curly-brace answer, so a correct answer in the wrong format counts as wrong. The authors say errors likely remain in some questions despite curation. The files read state an MIT license for the repository but say nothing separate about the dataset's terms.
How to use it well
Use it to probe the ceiling of a vision model, not to rank everyday performance: report all three numbers, and say whether tools were allowed. Run the subquestions alongside the main set, since they give partial signal on puzzles where the main score is zero. Pin the dataset version you test against, because answers were revised across versions. Expect to write your own inference loop and retry handling. If you need broad coverage of ordinary image understanding, this set is too small and too hard to tell you that.
Technical notes+
README.md is the whole product surface: it loads data via load_dataset('jonathan-roberts1/zerobench') with splits zerobench (100 rows) and zerobench_subquestions (334 rows), features question_id, question_text, question_images_decoded, question_answer, question_images, image_attribution. Images are shared between splits and also offered as one zip. The scoring snippet appends a step-by-step instruction asking for the answer in braces, extracts the last match of the regex \{(.*?)\} from the lowercased response, and marks correct when the extracted string equals the stripped, lowercased ground truth with equal length; inference is a commented-out placeholder. Rows are accumulated with pd.concat per question. LICENSE is MIT, copyright Jonathan Roberts. v3 lives on the main dataset branch, v2 on a v2 branch.
Observed
- License
- MIT (repository LICENSE file)
- Dataset size
- 100 main questions and 334 subquestions, sharing one image set
- Distribution
- Hugging Face dataset, loadable with the datasets library; images also as a zip download
- Metrics
- pass@1, pass@5 and pass^5 over five samplings per question
- Repository contents
- No code files in the tree; scoring code appears only as a README snippet
- Grading
- Exact case-insensitive match on the last curly-brace answer in the README snippet
- Contamination guard
- Canary string that is a superset of the BIG-bench canary
Read from README.md, LICENSE.
What it can do
Track official and externally reported model results across versions
Model version reports → Updated score records
Tags
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.