ZeroBench

Stated on the product site

Question set

The benchmark is built from a fixed set of 100 main questions plus 334 associated subquestions.

Leaderboard structure

Results are organized into three separate leaderboards: one for evaluations run by the benchmark's own team, one for scores that model developers or other labs reported themselves, and one preserving the original set of models tested at launch.

Scoring metrics

Each model's performance is reported using pass@1, pass@5, and pass^5 accuracy scores, along with average cost and token usage per question.

Tool-assisted runs

The site distinguishes model runs that used external tools, such as code execution or GUI interaction, from standard runs, marking the tool-assisted ones separately in its charts and tables.

Error analysis

The page also reports a categorization of common failure types behind incorrect model answers, derived from a multi-stage analysis process.

Not stated on the site

• It is not stated whether prospective evaluators can access the underlying question set and answer key directly, or only submit completed results for the maintainers to add.
• The page does not specify what licensing or usage terms govern the dataset or its images.
• It is not stated who funds or handles ongoing operation of the benchmark's public rankings beyond the individuals credited as paper authors.