Artificial Analysis
2 Tools · 276 Intel
Artificial Analysis is an independent AI benchmarking organization, founded in 2023 by Micah Hill-Smith and George Cameron, that runs its own evaluations of AI models and inference providers rather than relying on self-reported figures from AI labs. Its free public leaderboards and Intelligence Index have become widely referenced industry benchmarks for comparing model quality, speed, and cost. The company is backed by investors including Nat Friedman, Daniel Gross, and Andrew Ng.
Is this you? Sign in with X to claim this profile.
Tools
Independent benchmarks comparing AI models and providers on intelligence, coding, speed, latency, and price.
Artificial Analysis's platform for creating and running custom benchmarks against AI models on your own tasks.
Intel
Artificial Analysis benchmarks Exa's new Dynamic Highlights mode in auto and lite search types: Search Index 80 and 79, fewer searches and tokens per task, lower cost, but slower auto-mode latency. Lite costs $4 per 1k searches. Tested with its open Stirrup harness.
Artificial Analysis launches CritPt v2 with author-reviewed physics challenges and agentic code execution. GPT-6 Astra scores 83% (32% on v1), Claude Opus 5.5 78%. Scores are not comparable with v1, and the benchmark now has less headroom.
Mistral releases Large 4 in research preview: 38 on the Intelligence Index, 512k context, $1.36/$4.18 pricing, with 1T-parameter weights due end of October and a strong 82% on CyberGym-E2E-AA.
Xiaomi's newly released MiMo-V2.6-Pro, a 1.02T-parameter MoE model, becomes the top-scoring open-weights model on Artificial Analysis's Intelligence Index while undercutting rivals on price, landing on the cost-efficiency frontier.
Artificial Analysis covers Gemma 4's four sizes, Apache 2.0 license, doubled context, and native multimodality, with the 31B scoring 39 and using far fewer tokens than Qwen3.5 27B.
Artificial Analysis's evaluation of Meta's Muse Spark 1.2: 57 on the Intelligence Index for the xhigh reasoning variant, $1.25/$4.25 per million tokens with an 88% cache discount, roughly $0.40 per index task, a 1M-token context window, and text, image, speech and video input.
Artificial Analysis ranks new coding agents on its Coding Agent Index. Claude Sonnet 5.5 in Claude Code leads at 68 but costs $14.19 per task, while GPT-6.1 Sol in Codex scores 63 at $1.04.
Artificial Analysis compares Qwen3.5 27B (42) and Gemma 4 31B (39), noting a 2.5x token-efficiency gap, strong agentic results, weak hallucination avoidance, and single-GPU deployability.
Artificial Analysis finds Grok 4.3 scores 53, gains 321 Elo on GDPval-AA, and costs about 20% less to run than Grok 4.20 thanks to sharply lower token prices.
Artificial Analysis's Coding Agent Index v1.5 now reports safety refusal rates, finding Claude Fable 5.1 had the highest fallback rates in Claude Code (8.8%) and Devin Fusion (7.1%), meaning index scores partly reflect fallback-model performance.
Artificial Analysis rates Claude Haiku 5.5 at 43 on the Intelligence Index, with 33% on Terminal-Bench 4.0 and tiered pricing of $0.10/$0.50 up to 100k tokens, rising to $0.50/$2.50 beyond. AutomationBench is understated by over-refusal.
NVIDIA's Nemotron 3 Ultra (about 550B, 55B active) leads US open-weights models on the Artificial Analysis index, serves above 400 tokens per second, and was tested with turn-capped Terminal-Bench runs for latency.
Artificial Analysis evaluates Meta's Muse Spark 1.3: index 61 (xhigh) and 62 (max, limited preview), gains in Terminal-Bench and Tau3-Banking, and the lowest cost per task among comparable models.
Harvey LAB-AA v1.1 adds hallucination auditing and a three-judge panel to a 120-task private legal benchmark. Grok 4.7 leads at 9.4%, and over 60% of otherwise passing outputs contain material hallucinations.
Microsoft's new speech model reaches 2.0% word error rate on the AA-WER leaderboard, second only to Alibaba's Fun-Realtime-ASR-preview, at roughly 411x real time and $1.67 per 1,000 minutes. Coverage expands from 43 to 60 languages with diarization and word timestamps.
Claude Fable 5 launches at the top of the Artificial Analysis index, leading on five of ten benchmarks, with fallback to Opus 4.8 on about 8% of tasks and a 1M context window.
Artificial Analysis benchmarks GPT-5.6 Sol, Terra and Luna: Sol scores 59 at about $1.04 per task and leads its Coding Agent Index at 80 in Codex, with Terra and Luna at much lower cost.
Artificial Analysis ranks inclusionAI's 6B Ming-Image-0.1-Design as the best open-weights text-to-image model for UI/UX design, with leading open-weights layout and text rendering, while placing it #45 of 160 overall. It supports RGBA output and ships under the MIT license.
Artificial Analysis finds GPT-6 Sol ($2/$10) and Luna ($0.10/$0.50) at roughly half GPT-5.6 cost with level index scores, lower hallucination rates, a better Terminal-Bench, and regressions on GDPval-AA.
Artificial Analysis finds Gemini 3.5 Flash scores 55, tops the speed-intelligence frontier at over 280 tokens per second, and gains sharply on agentic evals, but costs 5.5x more to run than its predecessor.
Claude Sonnet 5 scores 53 on the Artificial Analysis index and matches Opus 4.8 on knowledge-work agents, but costs $2.29 per task because of heavier token and turn use. A promotional discount runs until September 1.
Artificial Analysis benchmarks Reflection's Beam, a 501B (23B active) Apache 2.0 model. It ranks second among US open-weight models, is token efficient, strong on agentic workflows, weak on terminal tasks, and hallucinates often. It scores 28 in Reflection's Mirror harness.
AA-Briefcase measures agents on long-horizon professional work: four multi-week scenarios in data science, product management, banking operations and industry strategy, 91 tasks producing spreadsheets, decks and memos, each graded on rubric checks plus pairwise analytical and presentation quality.
Microsoft's MAI-Transcribe-1 ranks fourth on Artificial Analysis word error rate at 3.0%, runs about 69x real time, supports 25 languages, and costs $6 per 1,000 minutes via Foundry.