Artificial Analysis
2 Tools · 276 Intel
Artificial Analysis is an independent AI benchmarking organization, founded in 2023 by Micah Hill-Smith and George Cameron, that runs its own evaluations of AI models and inference providers rather than relying on self-reported figures from AI labs. Its free public leaderboards and Intelligence Index have become widely referenced industry benchmarks for comparing model quality, speed, and cost. The company is backed by investors including Nat Friedman, Daniel Gross, and Andrew Ng.
Is this you? Sign in with X to claim this profile.
Tools
Independent benchmarks comparing AI models and providers on intelligence, coding, speed, latency, and price.
Artificial Analysis's platform for creating and running custom benchmarks against AI models on your own tasks.
Intel
Artificial Analysis benchmarks Meta's Muse Spark: 52 on its index with notable token efficiency and strong vision, but weaker agentic performance, and Meta's first release without open weights.
Artificial Analysis launches an independent leaderboard for ServiceNow's EnterpriseOps-Gym: multi-step, irreversible tasks across eight domains via MCP, graded by SQL checks on final state, where the top model reaches 51%.
Artificial Analysis benchmarks Ant Group's Ling 3.0 Tiny: 7.9B total and 1.3B active parameters, 262k context, MIT weights, scoring 25 on the Intelligence Index — well above the median for its size class — at 154 tokens/sec, with the tradeoff that it burns roughly 4x the median token budget per task
Artificial Analysis moves its Intelligence Index to v4.1: newer agentic benchmarks, a saturated test removed, and new cost, time and token per-task metrics with cached-input accounting, plus updated leaders and open-weights standings.
Artificial Analysis evaluates Thinking Machines' Inkling Small, a 276B MoE open-weights model scoring 40, within a point of the flagship at under a third of the size, with weaker agentic results.
Artificial Analysis and Harvey update the Legal Agent Benchmark to credit a task only if it satisfies every rubric item and has no material hallucination. Grok 4.7 leads at 9.4%, and most passing runs fail the gate.
Artificial Analysis benchmarks DeepSeek V4 Flash 0731: index 50 (+10), Terminal-Bench 79%, GDPval-AA 1559 Elo, same architecture and pricing, with weights due in coming weeks.
Artificial Analysis benchmarks NVIDIA Nemotron 3.5 Lightning (31.6B total, 3.6B active): index 24, Terminal-Bench 24%, fast serving, OpenMDW license and near-lossless NVFP4 weights.
Pre-launch GDPval-AA results: Opus 4.6 scores 1606 Elo, far ahead of GPT-5.2 xhigh, using 30 to 60% more tokens than Opus 4.5 and more turns, including image-viewer self-checks, making it the most expensive model run on the eval.
Artificial Analysis scores Alibaba's Qwen3.8 Max at 58 on its Intelligence Index — 13th of 187, well above the 35 median — with a 1M-token context and text, image and video input. The catch is throughput and verbosity: 20 output tokens/sec and 150M tokens burned on the index, about $1.13 per task.
Artificial Analysis ranks Cursor's Composer 2.5 third on its Coding Agent Index at 62, with a 35-point SWE-Bench-Pro gain over Composer 2 and per-task cost far below Claude Code and Codex.
Artificial Analysis adds GPT-6 Sol (Daybreak Blue), available only through OpenAI's Daybreak program, to the Cyber Index. It leads, scoring 32 points above the public GPT-6 Sol at $1.77 per task.
Artificial Analysis releases Intelligence Index v4.2, adding the AA-Briefcase and GDP.pdf evaluations, removing the saturated GPQA Diamond, and raising private held-out test weighting to 40% to resist gaming.
Artificial Analysis benchmarks Alibaba's proprietary Qwen3-Max-Thinking: Intelligence Index 40, GDPval-AA 1170, weak AA-Omniscience, text-only, tiered pricing, and behind several open-weights rivals.
Artificial Analysis launches the Cyber Index and an industry alliance. It combines CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA to measure how well agents find, reproduce, and patch vulnerabilities from source.
Artificial Analysis benchmarks Reflection's Beam, an open-weights 501B model. It scores 30 on the Intelligence Index, uses few tokens, does well on AutomationBench, weakly on Terminal-Bench, and hallucinates at 47%.
Artificial Analysis revises its medical long-context reasoning leaderboard with a new prompt and a three-model judge panel. Claude Sonnet 5.5 and GPT-6 Astra tie at 41.7%. Claude leads completeness, OpenAI leads accuracy, and per-task cost varies widely. Scores are not comparable to the previous ver
Artificial Analysis evaluates Claude Opus 5: index 61 at max, top GDPval-AA v2 and AA-Briefcase scores, 89% Terminal-Bench, and lower cost per task than Fable 5.
Artificial Analysis measures GPT-6.1 Sol: near-Astra Intelligence Index score, $0.72 per task at max effort versus $3.26 for Astra, a 12 point Terminal-Bench 4.0 gain, and lower hallucination rate.
Artificial Analysis benchmarks Apodex 1.1: Intelligence Index 44, GDPval-AA v2 Elo 1348, TerminalBench v2.1 at 70%, about 17k output tokens per task. It trails peers on knowledge reliability, with a 78.4% hallucination rate. Priced at $0.30/$3.00 per 1M tokens with a 256K context.
Artificial Analysis benchmarked Octen Search using its Stirrup agent harness: it scores 77 on the Search Index (third place), leads on speed at 16.9s per task and 0.2s per query, and is among the cheapest providers, driven by strong BrowseComp results.
Artificial Analysis launches the Decision Index v1.0 for models that return probabilities over fixed options in one call. It covers knowledge, hallucination, reasoning, and tool calling, and finds decision models overconfident.
AA-Robotics v0.1 gives frontier models control of a 6-axis robot arm across eight tabletop tasks. GPT-6 Astra leads with 69 but needs about 30 minutes and $16 per task.
Independent benchmarks of Alibaba's Qwen3.5-397B-A17B: #3 open-weights model, native vision, unified reasoning modes, big agentic gains on GDPval-AA and TerminalBench, and a still-high hallucination rate versus Kimi K2.5 and GLM-5.