Artificial Analysis
2 Tools · 299 Intel
Artificial Analysis is an independent AI benchmarking organization, founded in 2023 by Micah Hill-Smith and George Cameron, that runs its own evaluations of AI models and inference providers rather than relying on self-reported figures from AI labs. Its free public leaderboards and Intelligence Index have become widely referenced industry benchmarks for comparing model quality, speed, and cost. The company is backed by investors including Nat Friedman, Daniel Gross, and Andrew Ng.
Is this you? Sign in with X to claim this profile.
Tools
Independent benchmarks comparing AI models and providers on intelligence, coding, speed, latency, and price.
Artificial Analysis's platform for creating and running custom benchmarks against AI models on your own tasks.
Intel
Artificial Analysis scores Fable 5.1 at 66 on its Intelligence Index, the highest it has measured. Output use of about 1.7x Fable 5 lifts cost per task to $3.76 (20% more) despite the 75% cache read price cut. Effort settings span 11x in token use.
Artificial Analysis benchmarks Z AI's GLM-5.3-Flash (320B total, 18B active, MIT, 400k context): 57 on its Intelligence Index at $0.09 per task, 84.3% on Terminal-Bench v2.1, but only 28% accuracy on AA-Omniscience and heavy reasoning-token use.
Artificial Analysis benchmarks Meta's Muse Spark 1.2: 54 on its Intelligence Index, up 3 from 1.1, with gains concentrated in agentic work (GDPval-AA v2 Elo 1631, Terminal-Bench 2.1 at 80%). Pricing holds at $1.25/$4.25 per 1M tokens, roughly $0.40 per index task.
Artificial Analysis benchmarks Alibaba's Qwen3.8 Max (2.4T MoE, 95B active): 56 on the Intelligence Index, strong GDPval-AA gains, higher cost per task than Kimi K3, and a worse hallucination rate. Alibaba says open weights arrive next week, with first-party API pricing of $2.00 and $6.00 per millio
Artificial Analysis benchmarks Gemini 3.7 Flash: 56 on its Intelligence Index (up 4), about 340 output tokens per second, a Pareto-frontier position on time per task, and leading scores on AutomationBench-AA and AA-AnalystAgent. Includes pricing and a 1M-token context window.
Artificial Analysis introduces the Endpoint Accuracy Index, comparing serverless endpoints to a self-hosted reference on tool calling, scientific reasoning and long-context recall. Early results show output limits, tool-call parsing and context caps cause large accuracy gaps.
Artificial Analysis benchmarks DeepSeek V4 Pro 0813: Intelligence Index 53, agentic gains, 30% fewer output tokens, MIT weights, and a 264% blended price rise that leaves it barely on the cost Pareto frontier, close to V4 Flash.
Artificial Analysis patched its Intelligence Index to v4.1.1: tau3-Banking moves to v1.0.1, and HLE, AA-LCR and AA-Omniscience switch to a single newer grader model. Rankings hold, Claude Opus 5 stays first at 63, and most scores move under a point.
Artificial Analysis benchmarks Meta's Muse Glimmer, a 30B dense Apache 2.0 open-weights model scoring 35 on its index. It runs on a single GPU, but lags Qwen3.6 27B on GDPval-AA and shows an 82% hallucination rate, while leading its class on Tau3-Banking tool use.
Artificial Analysis benchmarks DeepSeek V4.1 Flash: Index score of 40, first on AutomationBench-AA at 69%, $0.30/$1.20 per 1M tokens, 1M context, MIT license. Very verbose at 89k tokens per task, yet still about 2.5x cheaper per task than V4 Pro.
Artificial Analysis launched a Search Index scoring seven search API providers inside its open-source Stirrup agent with one fixed model. It averages DeepSearchQA, BrowseComp and AA-Omniscience and reports cost and time per task. Parallel, Exa and Firecrawl lead, against a 33 no-search baseline.
Artificial Analysis added reward hacking corrections to its Coding Agent Index v1.4. Any passing Terminal-Bench v2.1 attempt found to be fetching published solutions gets zero. Tasks run with public internet and don't forbid lookup, so rates differ widely across agents and models. No rates are given
Artificial Analysis and Liquid AI benchmark small models on iPhone 17 Pro. Nanbeige4.2-3B and LFM2.5-2.6B tie at 63, but generation time varies 30x and peak memory 19x, and context caps reshape the ranking.
Artificial Analysis benchmarks NVIDIA's Nemotron 3.5 Lightning, a 31.6B-total, 3.6B-active hybrid Mamba-Transformer. It scores 24 on the Intelligence Index, gains sharply on Terminal-Bench and GDPval-AA, runs near 670 tokens/s, and ships under OpenMDW-1.1 with NVFP4 weights.
Artificial Analysis benchmarks OpenAI's GPT-Live-1, a full-duplex speech-to-speech model that delegates reasoning to a configurable backend text model. It tops the Speech to Speech Index at 81.5 and leads Tau Voice, but trails Grok on latency and audio reasoning.
fal post-trained MiniMax H3 into H3 Max, which now leads Artificial Analysis' image-to-video-with-audio board and sits third in text-to-video. It produces 5–15 second clips with native audio up to 768p at $0.04 per second, under the base H3 endpoint, with weights promised.
Artificial Analysis reports Grok 4.6 gaining sharply on AA-Briefcase, a private agentic knowledge-work benchmark. Confidence intervals overlap with Claude Fable 5, while cost per task is $4.42 versus $22.30, $17.79 for Opus 5 and $6.73 for Kimi K3.
Artificial Analysis launches AA-AnalystAgent, an 80-task benchmark for quantitative analysis on real spreadsheets and documents. It uses pass^5 as the headline metric, runs on the open-source Stirrup harness, and reports Claude Opus 5 at 54%, with GPT-5.5 and Claude Fable 5 close behind.
Artificial Analysis reports Cartesia's Sonic 3.6 at number one on both its Provider Voice and Controlled Voice speech arenas, with Elo scores, per-million-character pricing and generation speed compared against ElevenLabs, Speechify and Alibaba models.
Artificial Analysis ranks BreezeBlue's Breeze TTS 2 first among open-weights models in its Provider Voices arena at 1,215 Elo, 90 above Fish Audio S2 Pro. It trails on Controlled Voices, runs 45 chars/sec versus 102, and costs $34 per 1M characters hosted.
Artificial Analysis reports four Korean labs above 30 on its Intelligence Index. Motif 3 (open weights, 314B total, 13B active) scores 47 and Upstage's proprietary Solar Pro 4 scores 42, the highest outside the US and China. SK Telecom and LG AI Research follow with open models.
Meta Superintelligence Labs released Muse Voice Transcribe, its first streaming speech-to-text model. Artificial Analysis measures 3.1% WER at 0.16s after end of speech at $0.18 per audio hour, with 80ms chunking, 70 plus languages and hour-plus audio support.
Artificial Analysis plots Intelligence Index against Time per Task (output tokens divided by decode speed). OpenAI models hold the frontier under two minutes, and four of the six above that are Anthropic. Kimi K3 (max) takes about 11 minutes because of slow first-party API output.
Artificial Analysis now publishes measured cache hit rates and per-provider Cost per Task for open-weights models. GLM-5.3 (max) ranges from $0.92 to $8.68 per task, driven by cached-input pricing, with hit rates from 41% to 100% across 21 providers.