Compares frontier models on 161 scientific image tasks with three repeats each. Models with similar pass@3 differ widely on all-three-correct consistency, and cost per task varies about 99x, which affects model choice for visual agents.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
articleThinking Machines Releases Inkling 2 Small Highest Scoring Open Weights Model Outside Of Chinia Upon Weights Release It
articleNanbeige Has Released Nanbeige4 3 33b Scoring 20 On The Intelligence Index
articleGithub Copilot Fusion Orchestrates Anthropic And Openai Models To Balance Performance And Cost
articleGemini 4 Carbon New Number 1
