Artificial Analysis
2 Tools · 299 Intel
Artificial Analysis is an independent AI benchmarking organization, founded in 2023 by Micah Hill-Smith and George Cameron, that runs its own evaluations of AI models and inference providers rather than relying on self-reported figures from AI labs. Its free public leaderboards and Intelligence Index have become widely referenced industry benchmarks for comparing model quality, speed, and cost. The company is backed by investors including Nat Friedman, Daniel Gross, and Andrew Ng.
Is this you? Sign in with X to claim this profile.
Tools
Independent benchmarks comparing AI models and providers on intelligence, coding, speed, latency, and price.
Artificial Analysis's platform for creating and running custom benchmarks against AI models on your own tasks.
Intel
Artificial Analysis revises its medical long-context reasoning leaderboard with a new prompt and a three-model judge panel. Claude Sonnet 5.5 and GPT-6 Astra tie at 41.7%. Claude leads completeness, OpenAI leads accuracy, and per-task cost varies widely. Scores are not comparable to the previous ver
Artificial Analysis evaluates Claude Opus 5: index 61 at max, top GDPval-AA v2 and AA-Briefcase scores, 89% Terminal-Bench, and lower cost per task than Fable 5.
Artificial Analysis measures GPT-6.1 Sol: one point below GPT-6 Astra on its index, $0.72 per task at max effort versus $3.26, a 12 point Terminal-Bench 4.0 gain, lower hallucination rate, and a higher cache-read discount at unchanged token prices.
Artificial Analysis benchmarks Apodex 1.1: Intelligence Index 44, GDPval-AA v2 Elo 1348, TerminalBench v2.1 at 70%, about 17k output tokens per task. It trails peers on knowledge reliability, with a 78.4% hallucination rate. Priced at $0.30/$3.00 per 1M tokens with a 256K context.
Artificial Analysis updates its knowledge and hallucination benchmark with a refreshed 3,000-question set and a calibrated public set. Rank order holds, scores fall about 2 points on average, and Gemini 4 Carbon leads.
Artificial Analysis benchmarks Octen Search in its Stirrup agent harness with the model held fixed. Octen scores 77 on the Search Index (third), 86 on BrowseComp, averages 0.2s per query and costs $0.058 per task at $1 per 1k queries.
Artificial Analysis launches the Decision Index v1.0 for models that return probabilities over fixed options in one call. It covers knowledge, hallucination, reasoning, and tool calling, and finds decision models overconfident.
AA-Robotics v0.1 gives frontier models control of a 6-axis robot arm across eight tabletop tasks. GPT-6 Astra leads with 69 but needs about 30 minutes and $16 per task.
Artificial Analysis introduces its Progress Index, an item response theory scale covering models since 2022. Findings: frontier gains of about 65 to 70 points a year, open weights lagging proprietary models by a median of about 4 months, and new frontier highs roughly every four weeks in 2026.
Independent benchmarks of Alibaba's Qwen3.5-397B-A17B: #3 open-weights model, native vision, unified reasoning modes, big agentic gains on GDPval-AA and TerminalBench, and a still-high hallucination rate versus Kimi K2.5 and GLM-5.
Artificial Analysis publishes first AA-AgentPerf results for Meta's Muse Spark Open (816B, 42B active): about 150M tokens/s per MW on NVIDIA B300 and GB300, 70M on AMD MI355X, and 1.6-1.8x gains from tuned serving configs. Multi-node disaggregation is still untested.
Artificial Analysis benchmarks Gemini 3 Flash Preview: Intelligence Index 71, top AA-Omniscience accuracy, strong multimodal results, half the price of Gemini 3 Pro, but roughly 160M tokens used, one of the heaviest tested.
Artificial Analysis extends its Controlled Voice Arena to 9 new languages, finding Cartesia's Sonic models lead most non-English leaderboards while Inworld's Realtime TTS-2 tops Mandarin.
Artificial Analysis released v2.0 of its six occupation-mapped Capability Indexes, aligned to Intelligence Index v5.0. It raises weight on non-hallucination and agentic tool use, swaps in HLE-Diamond and AA-Omniscience v1.1, and reports Claude Opus 5.5 leading all six.
Artificial Analysis rates Kimi K2 Thinking (1T/32B, INT4) at 67 on its index, the open-weights leader, with strong tool-calling but the highest token usage recorded and a slow base endpoint versus a pricier turbo option.
NVIDIA's Nemotron 3 Ultra, 550B parameters with 55B active, scores 48 on the Artificial Analysis index, leads US open weights, and served over 300 tokens per second on a pre-release endpoint.
Artificial Analysis introduces AA-WER Streaming, measuring word error rate and latency after end of speech. Cartesia Ink-2 and ElevenLabs Scribe v2 lead accuracy, while Deepgram Flux is fastest.
Artificial Analysis says Intelligence Index v5 ships in late October. It adds Terminal-Bench Science and a new coding benchmark built on a private dataset, changes that affect how frontier models are ranked and how exposed coding scores are to training contamination.
Artificial Analysis benchmarks GPT-6 Astra: 67 on the Coding Agent Index in Codex using a third of GPT-5.6 Sol's tokens, hallucination down from 92% to 51%, but pricing rises to $10/$50 per million tokens, making general intelligence work 75% costlier per task.
Artificial Analysis introduces a Speech to Speech Index combining Big Bench Audio, Full Duplex Bench and tau-Voice. GPT-Realtime-2 leads at 77.2%, and agentic task completion stays under 53% for all models.
Artificial Analysis benchmarks Claude Opus 5.5 (max effort) at an Intelligence Index of 58, ranking near the top of 212 tracked models, while noting it runs expensive at $5.98 per task and unusually verbose at 260M output tokens versus peers.
Meta's Muse Spark 1.1 scores 51 on the Artificial Analysis index with a 1M context and low token use. Its hallucination rate fell mainly by abstaining more, while accuracy stayed flat.
Artificial Analysis reports Claude Opus 5.5 leading its Intelligence Index at 58, tying GPT-6 Astra on Terminal-Bench 4.0 at 59.6%. Anthropic cut prices 20% and cache reads to $0.20, but output token use rose 1.6x, leaving cost per task level with Opus 5.
Artificial Analysis evaluates Claude Fable 5.1: index 66 at max effort, record agentic knowledge work scores, cache read price cut to $0.25 per million, and higher cost per task from greater token use.