Artificial Analysis
2 Tools · 299 Intel
Artificial Analysis is an independent AI benchmarking organization, founded in 2023 by Micah Hill-Smith and George Cameron, that runs its own evaluations of AI models and inference providers rather than relying on self-reported figures from AI labs. Its free public leaderboards and Intelligence Index have become widely referenced industry benchmarks for comparing model quality, speed, and cost. The company is backed by investors including Nat Friedman, Daniel Gross, and Andrew Ng.
Is this you? Sign in with X to claim this profile.
Tools
Independent benchmarks comparing AI models and providers on intelligence, coding, speed, latency, and price.
Artificial Analysis's platform for creating and running custom benchmarks against AI models on your own tasks.
Intel
Artificial Analysis ranks Google's Gemini 4 Carbon first on Intelligence Index v5 and second on its Coding Agent Index. It reports cost per task and token use against Claude Opus 5.5, plus a lead on AA-Omniscience with a low hallucination rate.
Artificial Analysis finds Gemini 3.5 Flash scores 55, tops the speed-intelligence frontier at over 280 tokens per second, and gains sharply on agentic evals, but costs 5.5x more to run than its predecessor.
Claude Sonnet 5 scores 53 on the Artificial Analysis index and matches Opus 4.8 on knowledge-work agents, but costs $2.29 per task because of heavier token and turn use. A promotional discount runs until September 1.
Artificial Analysis benchmarks Reflection's Beam, a 501B (23B active) Apache 2.0 model. It ranks second among US open-weight models, is token efficient, strong on agentic workflows, weak on terminal tasks, and hallucinates often. It scores 28 in Reflection's Mirror harness.
Artificial Analysis reports Grok 4.6 gaining sharply on AA-Briefcase, a private benchmark of 91 long-horizon knowledge-work tasks, tying Claude Fable 5 within confidence intervals at $4.42 per task versus $22.30. The benchmark page details rubric and pairwise grading.
Microsoft's MAI-Transcribe-1 ranks fourth on Artificial Analysis word error rate at 3.0%, runs about 69x real time, supports 25 languages, and costs $6 per 1,000 minutes via Foundry.
Artificial Analysis benchmarks Meta's Muse Spark: 52 on its index with notable token efficiency and strong vision, but weaker agentic performance, and Meta's first release without open weights.
Artificial Analysis launches an independent leaderboard for ServiceNow's EnterpriseOps-Gym: multi-step, irreversible tasks across eight domains via MCP, graded by SQL checks on final state, where the top model reaches 51%.
Artificial Analysis benchmarks Ant Group's Ling 3.0 Tiny, an MIT-licensed 7.9B MoE with 1.3B active parameters. It scores 25 on the Intelligence Index, cuts hallucination sharply versus its predecessor, is modest on agentic work tasks, and uses very many output tokens.
Artificial Analysis moves its Intelligence Index to v4.1: newer agentic benchmarks, a saturated test removed, and new cost, time and token per-task metrics with cached-input accounting, plus updated leaders and open-weights standings.
Artificial Analysis launches VIALS-AA, its implementation of Handshake AI's scientific image reasoning benchmark. Results cover score, consistency across repeats, tool usage patterns, and cost per task across frontier models, with interactive agent traces.
Artificial Analysis evaluates Thinking Machines' Inkling Small, a 276B MoE open-weights model scoring 40, within a point of the flagship at under a third of the size, with weaker agentic results.
Artificial Analysis and Harvey add a hallucination gate to the Legal Agent Benchmark. Grok 4.7 leads at 9.4%, Muse Spark 1.3 falls from 26.7% to 8.9%, and GPT-6 Astra proves the most grounded, with per-task costs reported.
Artificial Analysis benchmarks DeepSeek V4 Flash 0731: index 50 (+10), Terminal-Bench 79%, GDPval-AA 1559 Elo, same architecture and pricing, with weights due in coming weeks.
Artificial Analysis benchmarks NVIDIA Nemotron 3.5 Lightning (31.6B total, 3.6B active): index 24, Terminal-Bench 24%, fast serving, OpenMDW license and near-lossless NVFP4 weights.
Pre-launch GDPval-AA results: Opus 4.6 scores 1606 Elo, far ahead of GPT-5.2 xhigh, using 30 to 60% more tokens than Opus 4.5 and more turns, including image-viewer self-checks, making it the most expensive model run on the eval.
GitHub's Copilot Fusion orchestrates Claude and GPT models at runtime. Artificial Analysis benchmarked its max setting: $5.59 per task versus $11.42 for Claude Code on Opus 5.5, about half the tokens, a Terminal-Bench 4.0 lead, and weaker SWE-Atlas-QnA results.
Artificial Analysis scores Alibaba's Qwen3.8 Max at 56 on its Intelligence Index, up 10 points, with GDPval-AA near Claude Fable 5. Cost rose to $1.14 per task and hallucination climbed from 23% to 40%. Alibaba says weights follow.
Artificial Analysis ranks Cursor's Composer 2.5 third on its Coding Agent Index at 62, with a 35-point SWE-Bench-Pro gain over Composer 2 and per-task cost far below Claude Code and Codex.
Artificial Analysis adds GPT-6 Sol (Daybreak Blue), available only through OpenAI's Daybreak program, to the Cyber Index. It leads, scoring 32 points above the public GPT-6 Sol at $1.77 per task.
Artificial Analysis releases Intelligence Index v4.2, adding the AA-Briefcase and GDP.pdf evaluations, removing the saturated GPQA Diamond, and raising private held-out test weighting to 40% to resist gaming.
Artificial Analysis benchmarks Alibaba's proprietary Qwen3-Max-Thinking: Intelligence Index 40, GDPval-AA 1170, weak AA-Omniscience, text-only, tiered pricing, and behind several open-weights rivals.
Artificial Analysis launches the Cyber Index and an industry alliance. It combines CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA to measure how well agents find, reproduce, and patch vulnerabilities from source.
Artificial Analysis benchmarks Reflection's Beam, an open-weights 501B model. It scores 30 on the Intelligence Index, uses few tokens, does well on AutomationBench, weakly on Terminal-Bench, and hallucinates at 47%.