Artificial Analysis
2 Tools · 299 Intel
Artificial Analysis is an independent AI benchmarking organization, founded in 2023 by Micah Hill-Smith and George Cameron, that runs its own evaluations of AI models and inference providers rather than relying on self-reported figures from AI labs. Its free public leaderboards and Intelligence Index have become widely referenced industry benchmarks for comparing model quality, speed, and cost. The company is backed by investors including Nat Friedman, Daniel Gross, and Andrew Ng.
Is this you? Sign in with X to claim this profile.
Tools
Independent benchmarks comparing AI models and providers on intelligence, coding, speed, latency, and price.
Artificial Analysis's platform for creating and running custom benchmarks against AI models on your own tasks.
Intel
Artificial Analysis benchmarks Upstage's Solar Pro 4: Intelligence Index 42 (up from 14), Terminal-Bench v2.1 at 57%, 512K context, higher price than Solar Pro 3, and a latency penalty. Hallucination rate falls mainly through abstaining more.
Artificial Analysis measured about 3,400 tokens/s median output on Gemma 4 31B using a private NVIDIA Groq 3 LPX endpoint, steady from 10k to 100k input tokens over 50 single-concurrency requests. The rack enters operation later this year.
Artificial Analysis benchmarks Ant Group's Ling 3.0 Flash, a 124B open weights reasoning model with 5B active parameters. It scores 38 on the Intelligence Index, leads on size-versus-intelligence, and lowers hallucination mostly by abstaining. Per-token pricing is low, but cost per task is less favo
Microsoft's MAI-Image-2.5-Pro, in preview on Foundry, takes first place on the Artificial Analysis image editing leaderboard and seventh in text to image. It costs roughly $108.5 per 1k images, against $48 for MAI-Image-2.5 and $20 for Flash.
Artificial Analysis' arena run puts Microsoft's MAI-Image-2.6-Preview first in image editing — ahead of its own 2.5-Pro, Reve 2.1 and GPT Image 2 — and second in text-to-image, leading 5 of 19 category boards including Retail & Ecommerce and Marketing. Available in MAI Playground and Foundry private
Artificial Analysis places Muse Spark 1.2 (xhigh) on the Intelligence Index vs cost-per-task Pareto frontier at $1.25/$4.25 per 1M tokens. It undercuts several GPT-5.6 and Kimi K3 configurations, while per-task cost rose from 1.1 because of heavier agentic token use.
Artificial Analysis benchmarks MBZUAI's K2 Horizon, an open weights MoE scoring 47 on its Intelligence Index. It beats MiniMax-M3 on GDPval-AA and tau-Banking, trails on GPQA Diamond and Humanity's Last Exam, and lowers hallucination by abstaining on 60% of questions. Pricing and license are pending
Artificial Analysis benchmarked Perplexity's Search API at three context sizes in a fixed agent harness. Medium scores 80 against 75 for Parallel and Brave, with the lowest inference cost per task. Quality plateaus between medium and high while searches and total cost per task fall.
Artificial Analysis benchmarks Grok Voice Transcribe 2.0: 2.7% streaming WER at 0.49s after end of speech, 2.3% non-streaming (5th of 59), priced at $0.20/hour streaming. Compared against ElevenLabs, Cartesia and Muse on accuracy, latency and cost.
Artificial Analysis evaluates Ant Group's Ling 3.0 Tiny: a 7.9B-total, 1.3B-active open MoE with 262K context and MIT license. It scores 25 on the Intelligence Index with a 30% hallucination rate, but uses heavy output tokens and trails on GDPval-AA.
Held out medical long context tiers, graded by panel, with Claude Fable 5 in front.
Artificial Analysis scores Agnes 2.5 Pro Beta at 49 on its Intelligence Index, up 9 points. The Agentic Index rises from 25 to 44, but output tokens per task double to 50k, and the Omniscience gain comes from abstention, with accuracy falling from 33% to 17%.
Human run speech model arena that scores preference and whether the tool call landed.
Artificial Analysis rebuilds its Text to Image Arena around 10 real-world use cases and 9 capabilities, from UI mockups to live-action film, with monthly-refreshed human-curated prompts. GPT Image 2 leads everywhere, with Nano Banana 2 the value pick.
Artificial Analysis benchmarks Google's Gemini 3.5 Transcribe at 2.6% WER (#5) and about 84x realtime for roughly $5 per 1,000 minutes, and the Live variant at 4.0% WER with 0.40s to first final transcript for about $9, compared against rival speech APIs.
Round 2 of Korea's Sovereign AI Foundation Model program cut the field to three: Upstage Solar Open 250B (Intelligence Index 37), SK Telecom A.X K2 (35) and LG K-EXAONE 2.0 (31). Each gets about 1,000 B200s for six months, with a further cut to two planned for early 2027.
Artificial Analysis launched Optima, which drafts tasks and grading rubrics from your description, examples or traces, runs them across models (or your own agent over HTTP), and reports score, cost and time per task. Grading is priced per criterion or match.
Artificial Analysis tests Ant Group's MIT-licensed Ling-3.0-flash-VL, a 124B MoE with 5.5B active parameters. It leads similar-size models on intelligence and hallucination but scores 0% on Terminal-Bench v4.0 and uses about 50k tokens per task.
Artificial Analysis on Muse Spark 1.2's cost efficiency: $0.40 per Intelligence Index task, with only Grok 4.5 and GPT-5.6 Sol at medium effort cheaper in its intelligence cluster, the rise from 1.1 driven by token usage rather than pricing.
Artificial Analysis breaks down Muse Spark 1.2's 3-point Index gain: concentrated in agentic evaluations, led by a 260-Elo jump on GDPval-AA v2, with small regressions on SciCode and Humanity's Last Exam.
Artificial Analysis's Endpoint Accuracy Index benchmarks provider endpoints for the same open weights against a self-hosted reference, using three evaluations. Scores range from 73% to 100%, pointing to quantization, KV-cache compression and context limits as hidden quality factors. Per-provider res
Artificial Analysis launches a Search Index that tests search APIs inside one agent harness with a fixed model, scoring quality, total cost per task and time. Parallel, Exa and Firecrawl lead, and better results cut token spend.
Artificial Analysis scores Qwen3.8 Max at 1739 Elo on GDPval-AA, ahead of Kimi K3 and tied with Fable 5 and GPT-5.6 Sol, with the 468-Elo gain partly driven by taking 64 turns per task against its predecessor's 14.
Artificial Analysis introduces an index that benchmarks serverless endpoints of open-weights models against a self-hosted reference on tool calling, reasoning and long context. Early results show output limits, tool-call handling and reduced reasoning tokens driving large gaps.