Artificial Analysis

Artificial Analysis evaluates Qwen3.8 Max: 56 on its Intelligence Index, GDPval-AA gains, a higher hallucination rate, and cost per task up sharply on agentic turns. Alibaba says the 2.4T MoE weights will be released the following week.

Artificial Analysis finds Qwen3.8 Max regressing 10 points on AA-Omniscience versus its predecessor: accuracy flat at ~31% while hallucination rises from 23% to 40%, attempting more questions it cannot answer.

Muse Spark 1.2 ranks fifth on GDPval-AA v2 at 1631 Elo, behind Claude Opus 5, GPT-5.6 Sol and Kimi K3 but a 260-point jump over Muse Spark 1.1's launch score.

Artificial Analysis benchmarks Meta's Muse Glimmer, a 30B dense Apache 2.0 open-weights model. It scores 35 on the Intelligence Index and fits one GPU, but trails same-class models on agentic knowledge work and hallucination while leading on Tau3-Banking tool use.

Artificial Analysis benchmarks Muse Spark 1.2: Intelligence Index 54, strong agentic knowledge work gains, about $0.40 per task at unchanged pricing, and a hallucination drop (38% to 28%) driven by abstaining more while accuracy fell to 38%.

Artificial Analysis benchmarks Upstage's Solar Pro 4: Intelligence Index 14 to 42, large agentic and long-context gains, doubled pricing and slower tasks. Its Omniscience improvement comes from abstaining (41% attempted), while accuracy stays near 19%.

Artificial Analysis evaluates NVIDIA's Nemotron 3.5 Lightning: a hybrid Mamba-Transformer with 3.6B active parameters that matches gpt-oss-120b on its index, gains 334 Elo on GDPval-AA, triples Terminal-Bench scores, and ships near-lossless NVFP4 weights.

Ling 3.0 Flash improves 48 points on AA-Omniscience over Ling 2.6, driven primarily by cutting the hallucination rate from 97% to 44%.

Artificial Analysis classifies 1,567 failed AA-AnalystAgent attempts across ten models into seven failure modes. Early anchoring on a wrong hypothesis leads at 57%, and models differ in whether they misread domain terms, override source evidence, or skip verification.

Artificial Analysis released Capability Indices v1.1 across six occupation domains. It adds agentic tool use from AutomationBench-AA, long-context tasks and AA-Briefcase, updates Terminal-Bench to v4.0, and reports Claude Fable 5.1 leading all six indices.

Artificial Analysis launches a Cyber Index combining CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA to score agents on finding and fixing vulnerabilities. Early results show Grok 4.7 and MiMo-V2.6-Pro leading, while refusals sharply penalize several frontier models.

Artificial Analysis benchmarks GPT-6.1 Sol: one point below GPT-6 Astra at $0.72 per task, unchanged $2/$10 pricing with a 95% cache discount, and a 12 point Terminal-Bench 4.0 gain.

Artificial Analysis released a release comparison tool for up to five models, showing every effort level on its Intelligence Index, domain Capability Indexes, individual benchmarks, itemized token pricing, cost per task, output speed and latency.

Artificial Analysis benchmarks Claude Opus 4.6: first on the Intelligence Index, wins on GDPval-AA, TerminalBench and CritPT, uses ~2x Opus 4.5's tokens, and introduces adaptive thinking with effort levels and a 1M context window.

Artificial Analysis scores Claude Sonnet 5.5 at 56 on its Intelligence Index, parity with Opus 5.5 on several agentic evals, but with the heaviest output token use it has measured, about 193k per task. Results come from a pre-release build with a known structured-output bug.

Artificial Analysis finds Kimi K3 scores second on its AA-Briefcase knowledge-work benchmark, but at about $10.57 per task, 83 turns and nearly an hour per run, with weaker presentation quality than analysis.

Artificial Analysis compares 20 guardrail models on WildGuardTest, ToxicChat and XSTest, covering F1, recall, specificity, over-refusal and latency across specialist classifiers, moderation APIs and prompted gpt-oss baselines.

Artificial Analysis benchmarks Gemini 3.8 Flash: 59 on its Intelligence Index (up 3), $0.58 per task, 2.5 minutes per task and 48k output tokens per task. Gains come mostly from agentic evals. Pricing is $0.75/$3.75 per million tokens through year end, with a 1M context window.

Artificial Analysis decomposes Opus 5.5 cost per Intelligence Index task: ~80% more tokens would raise it to $10.51, the 20% price cut brings it to $8.41, and cheaper cache reads bring it to $5.98, near Opus 5's $5.86.

Artificial Analysis finds Claude Sonnet 5.5 scoring 56 on the Intelligence Index and 64% on Terminal-Bench 4.0, held at $2/$10, with the heaviest token use measured and a structured-output bug fixed for release.

Alibaba's Wan 3.0 enters public preview on Model Studio as a single system for generating and instruction-editing video, accepting text, image, video, audio and document references. Artificial Analysis measures it first in video editing with audio, up from fifth for Wan 2.7.

Artificial Analysis places Alibaba's Qwen-Image-3.0-Pro sixth in image editing and ninth in text to image, up 83 and 48 Elo on the prior Pro model. Pricing starts at four cents per 1K image through Alibaba Cloud Model Studio.

Artificial Analysis benchmarks GPT-6 Astra: tied first with Claude Fable 5.1 on the Intelligence and Coding Agent indices at lower cost per task, hallucination down from 92% to 51%, leads Terminal-Bench v4.0, and pricing rises to $10/$50 per million tokens.

AA-AgentPerf puts NVIDIA's Vera Rubin NVL72 against GB300 on replayed coding-agent traces, reporting 2.8x to 5.7x throughput per megawatt across DeepSeek, GLM and Kimi models, and explains why agentic workloads stress KV-cache reuse.