By penalizing wrong answers rather than rewarding guessing, AA-Omniscience shows most models answer hard questions incorrectly more often than correctly. This exposes reliability risks that accuracy-only benchmarks hide.
Key takeaways · AI-distilled
Artificial Analysis argues standard benchmarks encourage hallucinationWhen a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.Full definition → because they never penalize wrong answers, pushing models and labs to attempt every question, and says factual knowledge is where guessing is least acceptable.
Claude 4.1 Opus led the Index at launch, followed by GPT-5.1 and Grok 4, yet even the leaders scored only slightly above 0. Anthropic's lead came from low hallucination; OpenAI's and xAI's from higher accuracy.
High accuracy did not mean low hallucination: Grok 4 led raw accuracy, but the GPT-5 models and Gemini 2.5 Pro, also near the top, missed the Index lead because they tended to guess rather than abstain. Claude 4.5 Haiku had the lowest hallucination rate.
The questions span 6 domains and 89 sub-topics such as Python data libraries, public policy and taxation; no model led every domain, with Claude 4.1 Opus ahead in Law, Software Engineering, and Humanities and Social Sciences.
Terms in this piece · Glossary
hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
postAA-Omniscience regresses 10 points from Qwen3.7 Max (+14 to +4), reversing its…Artificial Analysis
postSolar Pro 4's AA-Omniscience score improves from -53 to -1, however the improvemArtificialAnlys
postMuse Spark 1.2 continues Meta's AA-Omniscience pattern: the score rose from 18…Artificial Analysis