Artificial Analysis
2 Tools · 276 Intel
Artificial Analysis is an independent AI benchmarking organization, founded in 2023 by Micah Hill-Smith and George Cameron, that runs its own evaluations of AI models and inference providers rather than relying on self-reported figures from AI labs. Its free public leaderboards and Intelligence Index have become widely referenced industry benchmarks for comparing model quality, speed, and cost. The company is backed by investors including Nat Friedman, Daniel Gross, and Andrew Ng.
Is this you? Sign in with X to claim this profile.
Tools
Independent benchmarks comparing AI models and providers on intelligence, coding, speed, latency, and price.
Artificial Analysis's platform for creating and running custom benchmarks against AI models on your own tasks.
Intel
Artificial Analysis publishes first AA-AgentPerf results for Meta's Muse Spark Open (816B, 42B active): about 150M tokens/s per MW on NVIDIA B300 and GB300, 70M on AMD MI355X, and 1.6-1.8x gains from tuned serving configs. Multi-node disaggregation is still untested.
Artificial Analysis benchmarks Gemini 3 Flash Preview: Intelligence Index 71, top AA-Omniscience accuracy, strong multimodal results, half the price of Gemini 3 Pro, but roughly 160M tokens used, one of the heaviest tested.
Artificial Analysis extends its Controlled Voice Arena to 9 new languages, finding Cartesia's Sonic models lead most non-English leaderboards while Inworld's Realtime TTS-2 tops Mandarin.
Artificial Analysis rates Kimi K2 Thinking (1T/32B, INT4) at 67 on its index, the open-weights leader, with strong tool-calling but the highest token usage recorded and a slow base endpoint versus a pricier turbo option.
NVIDIA's Nemotron 3 Ultra, 550B parameters with 55B active, scores 48 on the Artificial Analysis index, leads US open weights, and served over 300 tokens per second on a pre-release endpoint.
Artificial Analysis introduces AA-WER Streaming, measuring word error rate and latency after end of speech. Cartesia Ink-2 and ElevenLabs Scribe v2 lead accuracy, while Deepgram Flux is fastest.
Artificial Analysis says Intelligence Index v5 lands in late October. It adds Terminal-Bench Science and a new coding benchmark built on a private dataset, which reduces contamination risk in frontier model comparisons.
Artificial Analysis benchmarks GPT-6 Astra: 67 on the Coding Agent Index in Codex using a third of GPT-5.6 Sol's tokens, hallucination down from 92% to 51%, but pricing rises to $10/$50 per million tokens, making general intelligence work 75% costlier per task.
Artificial Analysis introduces a Speech to Speech Index combining Big Bench Audio, Full Duplex Bench and tau-Voice. GPT-Realtime-2 leads at 77.2%, and agentic task completion stays under 53% for all models.
Artificial Analysis benchmarks Claude Opus 5.5 (max effort) at an Intelligence Index of 58, ranking near the top of 212 tracked models, while noting it runs expensive at $5.98 per task and unusually verbose at 260M output tokens versus peers.
Meta's Muse Spark 1.1 scores 51 on the Artificial Analysis index with a 1M context and low token use. Its hallucination rate fell mainly by abstaining more, while accuracy stayed flat.
Artificial Analysis's independent benchmarking shows Claude Opus 5.5 topping its Intelligence Index at 58, matching GPT-6 Astra on Terminal-Bench 4.0, and leading agentic knowledge-work evals, alongside a 20% price cut and cheaper cache reads.
Artificial Analysis evaluates Claude Fable 5.1: index 66 at max effort, record agentic knowledge work scores, cache read price cut to $0.25 per million, and higher cost per task from greater token use.
Artificial Analysis benchmarks Cognition's Devin Fusion, which runs a frontier lead model with a cheaper sidekick in parallel. It scores 68.1 on the Coding Agent Index, ahead of Claude Code with Opus 5.5 at 66.0, at about 31% lower cost per task and fewer minutes. Results are mixed across the three
Artificial Analysis load-tests 8x MI300X, H100 and H200 systems on DeepSeek R1 and Llama 4 Maverick. It maps the latency versus throughput tradeoff across concurrency levels and discloses AMD sponsorship and methodology.
Artificial Analysis's CyberGym-E2E-AA results show several frontier models safety-blocking most defensive memory-safety tasks, while cheaper capable models like GPT-6 Luna and MiMo-V2.6-Pro handle large-codebase bug hunts at a fraction of the cost.
Artificial Analysis benchmarks DeepSeek's new Search API as an agent web_search tool: index score 69 (17th of 27), cheapest per search, but weak on multi-hop BrowseComp, heavy 7k-token results, a 128-character query cap, and 5 searches per second.
Artificial Analysis benchmarks Xiaomi's MiMo-V2-Pro: 49 on its index, leading open-weights peers on GDPval-AA, low hallucination, 1M context, cheap pricing, but no released weights.
Artificial Analysis benchmarks Meta's Muse Spark 1.3: 61 (xhigh) and 62 (max, partner preview) on its Intelligence Index, gains concentrated in agentic evals, small regressions in long-context and accuracy, and the lowest cost per task among models scoring 59 or higher.
Artificial Analysis introduces AA-LCR, a benchmark of 100 questions over roughly 100k-token document sets. Top models score about 69%, and the results differ from needle-in-haystack tests and general reasoning rankings.
AA-AgentPerf replays real coding-agent trajectories to find how many concurrent agents a system sustains within speed and latency targets per megawatt. Initial NVIDIA and AMD results show Blackwell leading Hopper.
Artificial Analysis benchmarks put xAI's Grok 4.7 among the top agentic coding models, gaining +9 points on the Coding Agent Index and ranking 4th behind Claude Fable 5.1, GPT-6 Astra, and Claude Opus 5, though with higher token usage per task.
Artificial Analysis benchmarked Meta's Muse Wonder (52) and Muse Wonder Full (56) on its Intelligence Index, up from Muse Spark 1.3 at 48. Biggest gains came on agentic evals including Terminal-Bench 4.0, GDPval-AA and AA-Briefcase.
Artificial Analysis covers DeepSeek V4 Pro (1.6T, 49B active) and V4 Flash, scoring 52 and 47, leading open weights on agentic work but with hallucination rates above 90%.