Artificial Analysis
2 Tools · 276 Intel
Artificial Analysis is an independent AI benchmarking organization, founded in 2023 by Micah Hill-Smith and George Cameron, that runs its own evaluations of AI models and inference providers rather than relying on self-reported figures from AI labs. Its free public leaderboards and Intelligence Index have become widely referenced industry benchmarks for comparing model quality, speed, and cost. The company is backed by investors including Nat Friedman, Daniel Gross, and Andrew Ng.
Is this you? Sign in with X to claim this profile.
Tools
Independent benchmarks comparing AI models and providers on intelligence, coding, speed, latency, and price.
Artificial Analysis's platform for creating and running custom benchmarks against AI models on your own tasks.
Intel
Qwen3.8 Max costs $1.14 per Intelligence Index task, double its predecessor, despite lower per-token prices: turns per task on GDPval-AA rose from 14 to 64, so the agentic verbosity, not the pricing, drives the bill.
Artificial Analysis finds Qwen3.8 Max regressing 10 points on AA-Omniscience versus its predecessor: accuracy flat at ~31% while hallucination rises from 23% to 40%, attempting more questions it cannot answer.
Muse Spark 1.2 ranks fifth on GDPval-AA v2 at 1631 Elo, behind Claude Opus 5, GPT-5.6 Sol and Kimi K3 but a 260-point jump over Muse Spark 1.1's launch score.
Muse Glimmer's weaknesses concentrate in agentic evaluation: 953 Elo on GDPval-AA v2 against 1141 for Qwen3.6 27B, and 52% on Terminal-Bench v2.1 against 61%, with a hallucination rate of 82% on AA-Omniscience against 49%.
Artificial Analysis on Muse Spark 1.2's AA-Omniscience result: the score rose 18 to 22 driven by abstention again, hallucination rate down 10 points while accuracy slipped from 41% to 38%.
Artificial Analysis's numbers on Upstage's Solar Pro 4 show its AA-Omniscience score climbing from -53 to -1 almost entirely through refusal: attempts drop from 92% to 41% of questions and hallucinations from 88% to 24%, while accuracy sits unchanged around 19%.
NVIDIA's first Nemotron 3.5 model lands as a small open-weights release with a genuine agentic jump: 824 Elo on GDPval-AA v2, ahead of Nemotron 3 Super and gpt-oss-120b, and more than triple Nemotron 3 Nano on Terminal-Bench v2.1.
Ling 3.0 Flash improves 48 points on AA-Omniscience over Ling 2.6, driven primarily by cutting the hallucination rate from 97% to 44%.
A classification of 1,567 failing AA-AnalystAgent attempts across ten models into seven failure modes. Anchoring on a wrong early hypothesis is the most widespread at 57%, with models committing to a source early and defending it for the rest of the run.
Artificial Analysis updated its occupation-mapped Capability Indices to v1.1, adding Agentic Tool Use benchmarks from AutomationBench-AA and long-context tasks across Finance, Legal, Healthcare, Strategy, Engineering and Economics indices.
Artificial Analysis launches a Cyber Index and partner Alliance combining CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA to score how well AI agents audit, discover, patch, and reproduce vulnerabilities.
Artificial Analysis benchmarks GPT-6.1 Sol: one point below GPT-6 Astra at $0.72 per task, unchanged $2/$10 pricing with a 95% cache discount, and a 12 point Terminal-Bench 4.0 gain.
Artificial Analysis released a release comparison tool for up to five models, showing every effort level on its Intelligence Index, domain Capability Indexes, individual benchmarks, itemized token pricing, cost per task, output speed and latency.
Artificial Analysis benchmarks Claude Opus 4.6: first on the Intelligence Index, wins on GDPval-AA, TerminalBench and CritPT, uses ~2x Opus 4.5's tokens, and introduces adaptive thinking with effort levels and a 1M context window.
Artificial Analysis benchmarks Claude Sonnet 5.5: 56 on its Intelligence Index, parity with Opus 5.5 on several agentic evals, but the heaviest output token use it has measured, raising cost per task to $7.60.
Artificial Analysis finds Kimi K3 scores second on its AA-Briefcase knowledge-work benchmark, but at about $10.57 per task, 83 turns and nearly an hour per run, with weaker presentation quality than analysis.
Artificial Analysis compares 20 guardrail models on WildGuardTest, ToxicChat and XSTest, covering F1, recall, specificity, over-refusal and latency across specialist classifiers, moderation APIs and prompted gpt-oss baselines.
Artificial Analysis benchmarks Gemini 3.8 Flash: 59 on its Intelligence Index (up 3), $0.58 per task, 2.5 minutes per task and 48k output tokens per task. Gains come mostly from agentic evals. Pricing is $0.75/$3.75 per million tokens through year end, with a 1M context window.
Artificial Analysis decomposes Opus 5.5 cost per Intelligence Index task: ~80% more tokens would raise it to $10.51, the 20% price cut brings it to $8.41, and cheaper cache reads bring it to $5.98, near Opus 5's $5.86.
Artificial Analysis finds Claude Sonnet 5.5 scoring 56 on the Intelligence Index and 64% on Terminal-Bench 4.0, held at $2/$10, with the heaviest token use measured and a structured-output bug fixed for release.
Alibaba's Wan 3.0 enters public preview on Model Studio as a single system for generating and instruction-editing video, accepting text, image, video, audio and document references. Artificial Analysis measures it first in video editing with audio, up from fifth for Wan 2.7.
Artificial Analysis ranks Alibaba's Qwen-Image-3.0-Pro sixth for image editing and ninth for text to image, gains of 83 and 48 Elo over the previous Pro generation, at four cents per 1K image through Alibaba Cloud.
Artificial Analysis benchmarks GPT-6 Astra: tied first with Claude Fable 5.1 on the Intelligence and Coding Agent indices at lower cost per task, hallucination down from 92% to 51%, leads Terminal-Bench v4.0, and pricing rises to $10/$50 per million tokens.
AA-AgentPerf puts NVIDIA's Vera Rubin NVL72 against GB300 on replayed coding-agent traces, reporting 2.8x to 5.7x throughput per megawatt across DeepSeek, GLM and Kimi models, and explains why agentic workloads stress KV-cache reuse.