AI benchmark map

Tests models on editing code across several programming languages through Aider. Scores depend on the model configuration and the editing format used.

Sakana AI's benchmark of long-horizon algorithm engineering on hard combinatorial optimization problems from programming contests. Scores are contest-style ratings, so compare the time budget and iteration setup.

Tests whether models can write code that runs faster than expert reference implementations while remaining correct. Scores summarize speedups across many tasks, so read the aggregation method before comparing models.

Tests practical Python programming that combines multiple libraries. Generated solutions are checked against executable tests for tasks with detailed functional requirements.

Evaluates agent performance on coding tasks through Cline. Read the task set and agent configuration to understand what a reported result represents.

Programming contests provide algorithmic problem-solving tasks for model evaluations. Reported ratings depend on the contest selection, sampling budget, and evaluation protocol.

Cursor's internal suite of ambiguous, multi-file coding tasks drawn from real Cursor sessions. Versions change the problem mix, so compare scores within one version, alongside cost and tokens per task.

Evaluates coding agents on software engineering tasks. Check the evaluation version and agent configuration before comparing results across model releases.

Tests generated code with expanded test suites for programming benchmarks. The additional cases help reveal incorrect solutions that smaller test sets can miss.

Cognition's benchmark of hard, real open-source issues where coding agents must produce mergeable fixes. Epoch AI found too little public information to review it, so treat reported scores cautiously.

Ultra-long-horizon software engineering tasks spanning implementation, performance, scientific computing, visual reasoning, and AI research, with a twenty-hour budget per task. Compare the harness and budget before comparing agents.

Software optimization challenges where agents modify a program's code to make it substantially faster while staying correct. Compare the speedup threshold and evaluation harness alongside the pass rate.

Tests whether a model can complete small Python functions from prompts. Executable tests score correctness, with results affected by the number of sampled solutions.

Tests code generation using programming contest problems collected over time. Results depend on the evaluation window and the model’s sampling configuration.

Epoch AI's long-horizon coding benchmark. Models reimplement whole programs end to end from a specification without seeing the original source, so compare the task budget and harness alongside the score.

Measures local model performance using Apple’s MLX framework. Hardware, quantization, model size, and runtime settings affect the reported speed and resource use.

Tests whether agents can build complete Python repositories from natural-language specifications. Project test suites grade whether the generated code meets the requirements.

Tests agents on building programs from specifications. Executable evaluation checks whether the resulting implementation satisfies the required behavior across the benchmark’s tasks.

Evaluates models on React development tasks. Use it to explore framework-specific coding performance, while checking how solutions are tested and graded.

Scientist-curated research-coding problems across physics, chemistry, biology, and materials science. Each problem decomposes into subproblems, so check whether a reported score counts main problems or subproblems.

Tests whether coding agents can resolve real GitHub issues. Results depend on the benchmark split, repository environment, and agent harness used.

Tests coding agents on recent repository issues. The benchmark updates over time, so compare task dates and language coverage alongside reported scores.

Evaluates repository issue resolution with refreshed tasks. Its changing task pool helps track coding performance while reducing dependence on familiar public test cases.

Tests whether agents can complete practical tasks in a terminal. Compare the benchmark version and execution environment alongside the reported success rate.

Tests whether models can write working PyTorch code for unfamiliar, well-specified machine-learning problems under a fixed compute budget. Scores come from a non-agentic setup, so agent results are not comparable.

The American Invitational Mathematics Examination, a high-school olympiad qualifier with integer answers. Each year's paper is a separate edition; compare the same year, sample count, and sampling settings.

Tests generalization to unfamiliar tasks. Compare the specific ARC-AGI version and evaluation rules, because task formats and permitted interaction differ between releases.

Tests research mathematics from recent arXiv papers. Compare monthly editions, native agent harnesses, tool access, and final-answer grading; August 2026 prioritizes results that resolve earlier conjectures.

ByteDance Seed's contamination-resistant set of competition problems at or above late-AIME difficulty, each rewritten so the integer answer is as hard to guess as to derive.

Tests whether models recognize false mathematical conjectures instead of claiming to prove them. The August 2026 edition uses native agent harnesses and a revised 0–3 rubric, not binary accuracy.

Epoch AI's held-out, expert-written mathematics problems, ranging from advanced undergraduate through research level across tiers. Compare the tier, version, and sampling budget before comparing model scores.

Tests graduate-level reasoning in biology, chemistry, and physics. Diamond is an expert-validated subset; check the subset and sampling setup when comparing scores.

The Harvard-MIT Mathematics Tournament's February competition, with final-answer problems across algebra, combinatorics, geometry, and number theory. Each year is a separate edition; compare the same year and sample count.

Tests models on difficult questions across expert domains. Full, text-only, and tool-assisted evaluations have different scopes and should be compared separately.

Tests mathematical reasoning with difficult competition problems. Compare the problem-set edition and sampling budget alongside the percentage of correct final answers.

One thousand filtered factoid questions measuring parametric recall without tools. Headline accuracy also reflects a model's willingness to guess, so read abstention and calibration alongside the score.

Expert-authored research workflows from the life, physical, earth, mathematical, and engineering sciences, completed by agents in a terminal. Releases change the task set; compare the same release, harness, and trial count.

Tests agents on extended professional workflows across multiple domains. Compare the shared evaluation framework, agent harness, and task configuration alongside reported results.

Tests agents on business workflows across simulated applications. Grading checks the final system state for tasks spanning functions such as sales, support, and finance.

Tests models on choosing and calling functions correctly. Evaluation categories distinguish tool selection, argument construction, and more complex interaction patterns.

Tests whether agents can research questions on the live web and synthesize correct answers. Results depend on the browsing harness and the date of the run, so compare both alongside scores.

Compares development environments and sandboxes through practical tasks. Results describe the tested infrastructure and workflow, so they are not standalone model capability scores.

Puts a frontier model in control of a real Toyota Corolla's steering, throttle, and brakes through three MCP tools and scores progress along a fixed cone course. Compare attempt counts and harness latency before comparing progress.

Tests assistants on questions requiring reasoning, information retrieval, and tool use. Difficulty levels and available tools influence the reported completion rate.

OpenAI's evaluation of well-specified workplace tasks drawn from real occupations across nine economic sectors. Grading uses expert comparison against human deliverables, so read the win-rate definition before comparing models.

Tests agents on machine-learning engineering tasks drawn from competitions. Results reflect the full workflow, including data preparation, training, and submission quality.

Tests agents on tasks in desktop applications. Evaluation checks the resulting computer state, with performance affected by the model’s interaction setup and task budget.

Tests whether CLI agents can post-train small base models under a fixed compute budget, scored on seven target benchmarks. Epoch AI verified the design, but compare the budget and harness alongside results.

Tests how reusable skills affect agent task performance. Compare the skill configuration, model, and harness to distinguish improvements from changes in the setup.

Evaluates models on solving tasks with external tools. Check the tool collection, evaluation method, and execution conditions before comparing reported results.

Tests whether an agent can run a simulated vending machine business profitably and stay coherent over a full simulated year. Compare the run length and tool setup before comparing final balances.

Tests agents on tasks across realistic websites. Success is graded against the required outcome, covering workflows that involve navigation and changing application state.

Tests agents in tool-using conversations with simulated users. Success depends on following domain policies and reaching the required state across the interaction.

Cybersecurity capture-the-flag challenges measuring autonomous vulnerability discovery and exploitation in sandboxed environments. Compare the guidance setting and agent scaffold before comparing solve rates.

Tests whether agents can reproduce real software vulnerabilities. Agents receive a description and unpatched codebase, then produce executable proof-of-concept inputs.

Measures how far agents can take real, hardened vulnerabilities toward a working exploit, scored on a ladder of verifiable capability tiers. Few exploits are public, which limits contamination but complicates reproduction.

Tests whether agents can turn known vulnerabilities into working exploits. Evaluation runs in controlled environments and distinguishes exploitation from simply reproducing a crash.

Tests long-horizon security bug hunting in complex systems. Evaluation uses reproducible validation for tasks involving targets such as browser engines and the Linux kernel.

Tests visual discrimination, tracking, spatial perception, and pattern recognition. Direct vision answers and tool-assisted results use different setups and should be compared separately.

Tests understanding of real professional charts using expert-written questions. Charts span domains including science, finance, healthcare, engineering, and manufacturing.

Compares visual output through arena-style evaluations. Preference results depend on the task and judging setup, rather than a universal measure of design quality.

Compares computer-vision models on published evaluation tasks. Check the dataset, model configuration, and task-specific metric to interpret accuracy and performance results.

Tests difficult visual reasoning with separate main-question and subquestion scores. Compare sampling metrics and tool access before treating two reported results as equivalent.

Evaluates embedding models across multiple language tasks. Compare the benchmark version, language coverage, and task subset, especially when choosing models for retrieval.

Compares reranking models on ordering retrieved content. Check the dataset, candidate set, and ranking metric to judge relevance to your search workflow.

Evaluates audio models across supported speech and audio tasks. Results depend on the dataset and task-specific metric, so compare matching evaluation configurations.

Compares models using human preferences in paired interactions. Ratings reflect the voting population, category, and collection period, rather than objective correctness alone.

Publishes model evaluations and performance comparisons. Read each metric’s methodology and tested configuration when comparing capability, speed, cost, or human preference.

Epoch AI's hub of independently run benchmark results, the Epoch Capabilities Index, and benchmark reviews rating evaluations Verified or Flawed. Read each review before trusting a benchmark's headline numbers.

Compares inference performance across model and hardware configurations. Throughput and latency depend on the workload, serving stack, and resource allocation used.

Evaluates models on regularly refreshed tasks across several capabilities. Compare the release date and task category because the question set changes over time.

Estimates the length of software tasks a model completes correctly more often than not. Horizons are model estimates from a fixed task suite, so compare the suite version and confidence intervals.

Helps compare openly available models. Check the evidence behind each comparison, including the model version, evaluation source, and deployment configuration.

Helps explore model choices through published comparisons. Verify the source and date of each result before using it to select a model for your workload.

Evaluates agents through game-based competition. Outcomes depend on the game rules, opponents, and agent configuration rather than a general measure of model capability.

Tests how models respond to flawed or nonsensical prompts. It examines whether a model challenges the premise instead of confidently inventing an answer.

Evaluates agents on legal work. Read the task definitions and grading methodology to understand which parts of a legal workflow the results support.

Tests long-horizon decision-making in simulated sports betting markets. Agents build models, size bets, manage risk, and adapt their strategy across a football season.

Compares language-model training approaches under tight resource constraints. Results reflect the training rules and model-size budget as well as the final evaluation score.

Evaluates model responses to philosophical questions. Interpretation depends on the prompts and judging methodology, rather than a single objective notion of philosophical correctness.

Reports progress from one 141-hour GPT-6 Astra Minecraft run in Normal Survival with keepInventory enabled. These observations are not a standardized cross-model leaderboard or speed ranking.