A high accuracy figure can be misleading if it excludes the tasks that never returned an answer. In its o3 and o4-mini evaluation, ARC Prize found that high-reasoning runs completed only a subset of tasks. Earlier responses tended to be more accurate, suggesting the missing results were not a random sample.
That prevents treating accuracy among completed answers as performance on the whole benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →. The evaluator withheld those runs from leaderboard reporting rather than presenting the attractive partial results as reliable scores. The lesson is to check completion rates and timeout handling alongside accuracy, especially when comparing systems allowed different amounts of reasoning.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
