Artificial Analysis
2 Tools · 299 Intel
Artificial Analysis is an independent AI benchmarking organization, founded in 2023 by Micah Hill-Smith and George Cameron, that runs its own evaluations of AI models and inference providers rather than relying on self-reported figures from AI labs. Its free public leaderboards and Intelligence Index have become widely referenced industry benchmarks for comparing model quality, speed, and cost. The company is backed by investors including Nat Friedman, Daniel Gross, and Andrew Ng.
Is this you? Sign in with X to claim this profile.
Tools
Independent benchmarks comparing AI models and providers on intelligence, coding, speed, latency, and price.
Artificial Analysis's platform for creating and running custom benchmarks against AI models on your own tasks.
Intel
Artificial Analysis benchmarks Grok 4.6: 61 on the Intelligence Index, strong GDPval-AA, tau3-Banking and Terminal-Bench results, $0.84 per task, and turn-efficient agent runs. Speed is below average at 71 tokens per second with a long time to first token.
Artificial Analysis benchmarks Ling 3.1 Flash, a 560B/25B-active reasoning model with 1M context. Its index score doubles to 41, hallucination falls to 38%, and cost is $0.99 per task. Weights are promised but not yet released.
Artificial Analysis benchmarks Exa's new Dynamic Highlights mode in auto and lite search types: Search Index 80 and 79, fewer searches and tokens per task, lower cost, but slower auto-mode latency. Lite costs $4 per 1k searches. Tested with its open Stirrup harness.
Artificial Analysis launches CritPt v2 with author-reviewed physics challenges and agentic code execution. GPT-6 Astra scores 83% (32% on v1), Claude Opus 5.5 78%. Scores are not comparable with v1, and the benchmark now has less headroom.
Mistral releases Large 4 in research preview: 38 on the Intelligence Index, 512k context, $1.36/$4.18 pricing, with 1T-parameter weights due end of October and a strong 82% on CyberGym-E2E-AA.
Artificial Analysis reports Xiaomi's MiMo-V2.6-Pro as the top open-weights model on its Intelligence Index at 46, up from 26. The MoE model has 1.02T total and 42B active parameters, costs $0.13 per index task, and sits on the cost-per-task Pareto frontier.
Artificial Analysis covers Gemma 4's four sizes, Apache 2.0 license, doubled context, and native multimodality, with the 31B scoring 39 and using far fewer tokens than Qwen3.5 27B.
Artificial Analysis benchmarks Gemini 4 Carbon: first on Intelligence Index v5, second on its Coding Agent Index, with lower cost per task than Claude Opus 5.5. Hallucination rate rises to 11% versus Argon. Pricing is $4/$20 per 1M tokens, 1M context, with rollout limited to selected users.
Artificial Analysis benchmarks Phonic 1 (High) as #1 on its Speech to Speech Index at 87.1, narrowly ahead of Grok Voice 3.0. It leads Tau Voice agentic scoring, is #8 on arena preference, and costs $4.20 per hour of input audio.
Artificial Analysis benchmarks Meta's Muse Spark 1.2: 54 on the Intelligence Index (up from 51), GDPval-AA Elo of 1631, about $0.40 per task at unchanged $1.25/$4.25 pricing, 1M context. Hallucination fell to 28% mainly because attempt rate dropped to 67%. Since superseded by 1.3.
Artificial Analysis's Coding Agent Index ranks model and harness combinations. Sonnet 5.5 in Claude Code leads at 68 and $14.19 per task. Gemini 4 Argon in Antigravity CLI scores 64 at $5.84 (promotional pricing), and GPT-6.1 Sol in Codex scores 63 at $1.04.
Artificial Analysis compares Qwen3.5 27B (42) and Gemma 4 31B (39), noting a 2.5x token-efficiency gap, strong agentic results, weak hallucination avoidance, and single-GPU deployability.
Artificial Analysis finds Grok 4.3 scores 53, gains 321 Elo on GDPval-AA, and costs about 20% less to run than Grok 4.20 thanks to sharply lower token prices.
Artificial Analysis added safety refusal reporting to Coding Agent Index v1.5. Claude Fable 5.1 had the highest fallback rates, 8.8% in Claude Code and 7.1% in Devin Fusion, so those scores partly reflect fallback models. Harness and effort settings also affect rates.
Artificial Analysis rates Claude Haiku 5.5 at 43 on the Intelligence Index, with 33% on Terminal-Bench 4.0 and tiered pricing of $0.10/$0.50 up to 100k tokens, rising to $0.50/$2.50 beyond. AutomationBench is understated by over-refusal.
NVIDIA's Nemotron 3 Ultra (about 550B, 55B active) leads US open-weights models on the Artificial Analysis index, serves above 400 tokens per second, and was tested with turn-capped Terminal-Bench runs for latency.
Artificial Analysis benchmarks Ling 3.1 Flash 1019, a 560B (25B active) MIT-licensed open-weights model with 1M context. It scores 44 on Intelligence Index v4.3, with agentic gains but 145k output tokens per task and $1.49 cost per task.
Artificial Analysis evaluates Meta's Muse Spark 1.3: index 61 (xhigh) and 62 (max, limited preview), gains in Terminal-Bench and Tau3-Banking, and the lowest cost per task among comparable models.
Harvey LAB-AA v1.1 adds hallucination auditing and a three-judge panel to a 120-task private legal benchmark. Grok 4.7 leads at 9.4%, and over 60% of otherwise passing outputs contain material hallucinations.
Microsoft's new speech model reaches 2.0% word error rate on the AA-WER leaderboard, second only to Alibaba's Fun-Realtime-ASR-preview, at roughly 411x real time and $1.67 per 1,000 minutes. Coverage expands from 43 to 60 languages with diarization and word timestamps.
Claude Fable 5 launches at the top of the Artificial Analysis index, leading on five of ten benchmarks, with fallback to Opus 4.8 on about 8% of tasks and a 1M context window.
Artificial Analysis benchmarks GPT-5.6 Sol, Terra and Luna: Sol scores 59 at about $1.04 per task and leads its Coding Agent Index at 80 in Codex, with Terra and Luna at much lower cost.
Artificial Analysis ranks inclusionAI's 6B Ming-Image-0.1-Design as the best open-weights text-to-image model for UI/UX design, with leading open-weights layout and text rendering, while placing it #45 of 160 overall. It supports RGBA output and ships under the MIT license.
Artificial Analysis finds GPT-6 Sol ($2/$10) and Luna ($0.10/$0.50) at roughly half GPT-5.6 cost with level index scores, lower hallucination rates, a better Terminal-Bench, and regressions on GDPval-AA.