AI benchmark map

Limitations. Public solutions, reward hacking and task-specific tuning can inflate results. Earlier editions and vendor-reported runs do not establish the current reviewed protocol. Terminal completion is different from repository issue repair.

Reviewed 2026-10-19

Limitations. Samaritan Research reports false-negative grading defects in at least 23 of 113 v1.1 tasks. A subsequent repair has not been verified in this review, so affected results need caution.

Reviewed 2026-10-19

Limitations. Reflection co-developed V2. Public tasks remain exposed and the full-set chart is near ceiling. Error-bar construction is not established in this review. Publisher pages disagree on correction counts. Private, HARD, V1 and generic Pro results do not establish this protocol.

Reviewed 2026-10-19

Limitations. OpenAI stopped reporting Verified after finding task defects and memorized solutions. Its 59.4% defect finding concerns 138 selected difficult tasks, not the full 500-task set. Imported Epoch runs have their own protocol.

Reviewed 2026-10-19

Limitations. Match the problem window to the model cutoff and retain the dataset revision. Official errata documents grading and test defects. An unknown training cutoff prevents a strong contamination-free claim.

Reviewed 2026-10-19

Tests models on editing code across several programming languages through Aider. Scores depend on the model configuration and the editing format used.

Sakana AI's benchmark of long-horizon algorithm engineering on hard combinatorial optimization problems from programming contests. Scores are contest-style ratings, so compare the time budget and iteration setup.

Tests whether models can write code that runs faster than expert reference implementations while remaining correct. Scores summarize speedups across many tasks, so read the aggregation method before comparing models.

Tests practical Python programming that combines multiple libraries. Generated solutions are checked against executable tests for tasks with detailed functional requirements.

Evaluates agent performance on coding tasks through Cline. Read the task set and agent configuration to understand what a reported result represents.

Programming contests provide algorithmic problem-solving tasks for model evaluations. Reported ratings depend on the contest selection, sampling budget, and evaluation protocol.

Cursor's internal suite of ambiguous, multi-file coding tasks drawn from real Cursor sessions. Versions change the problem mix, so compare scores within one version, alongside cost and tokens per task.

Tests generated code with expanded test suites for programming benchmarks. The additional cases help reveal incorrect solutions that smaller test sets can miss.

Cognition's benchmark of hard, real open-source issues where coding agents must produce mergeable fixes. Epoch AI found too little public information to review it, so treat reported scores cautiously.

Ultra-long-horizon software engineering tasks spanning implementation, performance, scientific computing, visual reasoning, and AI research, with a twenty-hour budget per task. Compare the harness and budget before comparing agents.

Software optimization challenges where agents modify a program's code to make it substantially faster while staying correct. Compare the speedup threshold and evaluation harness alongside the pass rate.

Tests whether a model can complete small Python functions from prompts. Executable tests score correctness, with results affected by the number of sampled solutions.

Epoch AI's long-horizon coding benchmark. Models reimplement whole programs end to end from a specification without seeing the original source, so compare the task budget and harness alongside the score.

Evaluates models on Apple MLX API knowledge and coding across 441 bundled v2 questions. Check the dataset version, question filters, answer model, and judge configuration before comparing accuracy.

Tests whether agents can build complete Python repositories from natural-language specifications. Project test suites grade whether the generated code meets the requirements.

Tests agents on building programs from specifications. Executable evaluation checks whether the resulting implementation satisfies the required behavior across the benchmark’s tasks.

Evaluates models on React development tasks. Use it to explore framework-specific coding performance, while checking how solutions are tested and graded.

Scientist-curated research-coding problems across physics, chemistry, biology, and materials science. Each problem decomposes into subproblems, so check whether a reported score counts main problems or subproblems.

Tests coding agents on recent repository issues. The benchmark updates over time, so compare task dates and language coverage alongside reported scores.

Repository issue repair across multiple programming languages. Compare task selection, agent scaffold and evaluation settings; this suite is distinct from SWE-bench Verified.

Historical V1 repository issue tasks from Scale AI. Preserve the reported split, agent harness and execution budget. V1 scores do not establish performance on the revised V2 protocol.

Evaluates repository issue resolution with refreshed tasks. Its changing task pool helps track coding performance while reducing dependence on familiar public test cases.

Tests whether models can write working PyTorch code for unfamiliar, well-specified machine-learning problems under a fixed compute budget. Scores come from a non-agentic setup, so agent results are not comparable.

Limitations. A strong ARC result does not establish workplace competence. Keep standard and provider-adapter runs apart.

Reviewed 2026-10-15

Limitations. The static 198-question subset has limited statistical power and exposure risk. Its expert filtering does not make it representative of scientific practice. A Google-proof collection process does not prove present-day contamination resistance.

Reviewed 2026-10-19

Limitations. Difficulty is specialized; tier, dataset release, tools and grader must match before comparing scores.

Reviewed 2026-10-15

XingChen's model-card claim labeled AA.LCR. The publisher does not link a benchmark definition or identify its version. Treat this result as source-specific evidence.

The American Invitational Mathematics Examination, a high-school olympiad qualifier with integer answers. Each year's paper is a separate edition; compare the same year, sample count, and sampling settings.

Tests research mathematics from recent arXiv papers. Compare monthly editions, native agent harnesses, tool access, and final-answer grading; August 2026 prioritizes results that resolve earlier conjectures.

ByteDance Seed's contamination-resistant set of competition problems at or above late-AIME difficulty, each rewritten so the integer answer is as hard to guess as to derive.

Tests whether models recognize false mathematical conjectures instead of claiming to prove them. The August 2026 edition uses native agent harnesses and a revised 0–3 rubric, not binary accuracy.

Tests graph traversal and parent retrieval across long text contexts without tools. Scores depend on context length and dataset fixes; synthetic graph accuracy does not establish performance on real documents or codebases.

The Harvard-MIT Mathematics Tournament's February competition, with final-answer problems across algebra, combinatorics, geometry, and number theory. Each year is a separate edition; compare the same year and sample count.

Tests models on difficult questions across expert domains. Full, text-only, and tool-assisted evaluations have different scopes and should be compared separately.

Tests mathematical reasoning with difficult competition problems. Compare the problem-set edition and sampling budget alongside the percentage of correct final answers.

One thousand filtered factoid questions measuring parametric recall without tools. Headline accuracy also reflects a model's willingness to guess, so read abstention and calibration alongside the score.

Expert-authored research workflows from the life, physical, earth, mathematical, and engineering sciences, completed by agents in a terminal. Releases change the task set; compare the same release, harness, and trial count.

Limitations. AST matching, state checks and required-call checks measure different failure modes. Native function calling and prompted interfaces differ. The current V4 methods specify weighted categories, while the leaderboard summary says unweighted; preserve the exact revision and do not recompute a headline from that summary.

Reviewed 2026-10-19

Limitations. Not actively updated after September 8, 2026. Results above 16 human-hours are unreliable for this suite.

Reviewed 2026-10-15

Limitations. Full and offline datasets, task releases, step budgets, effort and tool settings stay separate. Provider effort labels do not imply equal compute. Exact execution backends and measurement dates are unreported; Muse Spark task denominators remain unresolved.

Reviewed 2026-10-19

Tests agents on extended professional workflows across multiple domains. Compare the shared evaluation framework, agent harness, and task configuration alongside reported results.

Professional workflows across law, consulting, and investment banking, graded against expert rubrics. Version 1.1 changes tasks and judges to penalize hedged answers; compare editions, effort settings, and scoring metrics.

Tests agents on business workflows across simulated applications. Grading checks the final system state for tasks spanning functions such as sales, support, and finance.

Agent evaluation whose primary metric requires success in all three trials. A three-run average is a different aggregation.

Tests whether agents can research questions on the live web and synthesize correct answers. Results depend on the browsing harness and the date of the run, so compare both alongside scores.

XingChen's model-card claim labeled DeepresearchBII, evaluated with OpenCode and Exa. Its benchmark definition and grader are unconfirmed. It is not merged with other deep-research benchmark results.

Compares development environments and sandboxes through practical tasks. Results describe the tested infrastructure and workflow, so they are not standalone model capability scores.

Puts a frontier model in control of a real Toyota Corolla's steering, throttle, and brakes through three MCP tools and scores progress along a fixed cone course. Compare attempt counts and harness latency before comparing progress.

Tests assistants on questions requiring reasoning, information retrieval, and tool use. Difficulty levels and available tools influence the reported completion rate.

OpenAI's evaluation of well-specified workplace tasks drawn from real occupations across nine economic sectors. Grading uses expert comparison against human deliverables, so read the win-rate definition before comparing models.

Tests agents on machine-learning engineering tasks drawn from competitions. Results reflect the full workflow, including data preparation, training, and submission quality.

Tests whether CLI agents can post-train small base models under a fixed compute budget, scored on seven target benchmarks. Epoch AI verified the design, but compare the budget and harness alongside results.

Tests how reusable skills affect agent task performance. Compare the skill configuration, model, and harness to distinguish improvements from changes in the setup.

Tests sustained planning in interactive fiction games, with separate runs with and without clues. Epoch identified progress-scoring and task-access defects, so reported game progress is not equivalent to verified completion.

Evaluates models on solving tasks with external tools. Check the tool collection, evaluation method, and execution conditions before comparing reported results.

Tests whether an agent can run a simulated vending machine business profitably and stay coherent over a full simulated year. Compare the run length and tool setup before comparing final balances.

Tests agents on tasks across realistic websites. Success is graded against the required outcome, covering workflows that involve navigation and changing application state.

Tests agents in tool-using conversations with simulated users. Success depends on following domain policies and reaching the required state across the interaction.

Sierra agent evaluation with revised tasks, knowledge and voice settings. Compare task selection and simulator configuration before comparing scores.

Limitations. Reproduction does not establish novel vulnerability discovery or exploit development. Network access and exposed patches can leak answers. Imported vendor-reported rows with unknown harnesses are not certified as the reviewed protocol.

Reviewed 2026-10-19

Cybersecurity capture-the-flag challenges measuring autonomous vulnerability discovery and exploitation in sandboxed environments. Compare the guidance setting and agent scaffold before comparing solve rates.

Measures how far agents can take real, hardened vulnerabilities toward a working exploit, scored on a ladder of verifiable capability tiers. Few exploits are public, which limits contamination but complicates reproduction.

Tests whether agents can turn known vulnerabilities into working exploits. Evaluation runs in controlled environments and distinguishes exploitation from simply reproducing a crash.

Tests long-horizon security bug hunting in complex systems. Evaluation uses reproducible validation for tasks involving targets such as browser engines and the Linux kernel.

Limitations. A shared dataset name does not establish comparable runs. The imported EdgeCrafter claims lack verified evaluator revisions, executed configurations, evaluated counts and measurement dates. They remain attributed publisher results without positions.

Reviewed 2026-10-19

Limitations. Public tasks and model judging limit small-score comparisons. This does not measure object detection or design preference. Imported tools-enabled vendor results do not certify the direct MLLM setup.

Reviewed 2026-10-19

Tests understanding of real professional charts using expert-written questions. Charts span domains including science, finance, healthcare, engineering, and manufacturing.

Compares visual output through arena-style evaluations. Preference results depend on the task and judging setup, rather than a universal measure of design quality.

Image segmentation evaluation from Technology Innovation Institute. Inspect the evaluated system; a detector combined with SAM2 is not the bare detector.

Compares computer-vision models on published evaluation tasks. Check the dataset, model configuration, and task-specific metric to interpret accuracy and performance results.

Tests difficult visual reasoning with separate main-question and subquestion scores. Compare sampling metrics and tool access before treating two reported results as equivalent.

Limitations. Public-task scores can reflect training exposure. Private RTEB data limits independent reproduction, and its launch report flags QA-derived keyword overlap. Imported English v2 results do not represent RTEB or complete RAG answer quality.

Reviewed 2026-10-19

Compares reranking models on ordering retrieved content. Check the dataset, candidate set, and ranking metric to judge relevance to your search workflow.

Limitations. Dataset weighting and language mix matter. Batch and streaming results are different operating conditions.

Reviewed 2026-10-15

Speech recognition evaluated by word error rate, where lower is better. Compare test splits, normalizers and aggregation; publisher submissions are not independent leaderboard placements.

Evaluates audio models across supported speech and audio tasks. Results depend on the dataset and task-specific metric, so compare matching evaluation configurations.

Limitations. Publisher weights, private tasks and mixed graders shape the index. Inference budgets differ, and component uncertainty does not establish composite uncertainty. Primarily English and text-based results do not establish broad multimodal quality. Unversioned imports are not certified as v4.3.2.

Reviewed 2026-10-19

Limitations. Single runs and averages over different measured suites limit overall comparisons. Research uses rubrics and Design uses visual and structural grading; grader validation is not disclosed. V1 contamination claims do not certify V2. Measurement dates are not reported.

Reviewed 2026-10-19

Compares inference performance across model and hardware configurations. Throughput and latency depend on the workload, serving stack, and resource allocation used.

Helps compare openly available models. Check the evidence behind each comparison, including the model version, evaluation source, and deployment configuration.

Helps explore model choices through published comparisons. Verify the source and date of each result before using it to select a model for your workload.

Limitations. Does not establish investment performance. Preserve version, tool access and split.

Reviewed 2026-10-15

Limitations. A calibrated model judge scores private held-out questions. All-pass and partial credit are different metrics.

Reviewed 2026-10-15

Evaluates agents through game-based competition. Outcomes depend on the game rules, opponents, and agent configuration rather than a general measure of model capability.

Tests how models respond to flawed or nonsensical prompts. It examines whether a model challenges the premise instead of confidently inventing an answer.

Evaluates agents on legal work. Read the task definitions and grading methodology to understand which parts of a legal workflow the results support.

Evaluates responses to clinician-authored healthcare tasks using physician-written rubrics and an LLM judge. Epoch found substantial rubric and scoring errors, so scores should not establish clinical reliability or safety.

Instruction-following evaluation from Ai2. Compare prompt or instruction scoring, strictness and sampling settings before comparing results across different models.

Tests long-horizon decision-making in simulated sports betting markets. Agents build models, size bets, manage risk, and adapt their strategy across a football season.

Compares short stories that must incorporate specified elements using model-judged pairwise ratings. Evaluator changes and undocumented score bridging limit comparability over time; the current ratings are separate from archived absolute grades.

Compares language-model training approaches under tight resource constraints. Results reflect the training rules and model-size budget as well as the final evaluation score.

Evaluates model responses to philosophical questions. Interpretation depends on the prompts and judging methodology, rather than a single objective notion of philosophical correctness.

Reports progress from one 141-hour GPT-6 Astra Minecraft run in Normal Survival with keepInventory enabled. These observations are not a standardized cross-model leaderboard or speed ranking.

Limitations. Original MMMU test answers are public; inspect the Pro release separately. The project leaderboard is dated September 2025. Keep standard and vision-only settings separate.

Reviewed 2026-10-15

Limitations. The site abstract and results artifact disagree on some scores. Public answers, embedding-based answer matching and model-judge parsing affect interpretation. Dataset license is noncommercial.

Reviewed 2026-10-19

Limitations. Separate subtitle text from raw audio, frame budgets and thinking settings. Public data does not prove freedom from contamination. Dataset use is restricted to academic research.

Reviewed 2026-10-19

Limitations. Preference is not objective correctness. Prompt mix and correlated votes affect interpretation. Imported Elo rows without a reported version are not certified as these editions.

Reviewed 2026-10-15

Limitations. Does not measure video understanding. Compare the same input format and generation conditions.

Reviewed 2026-10-15