AA-Omniscience v1.1 refreshes the hallucination benchmark

Artificial Analysis has cut its AA-Omniscience benchmark from 6,000 questions to 3,000 and published a new public sample, and the leaderboard mostly survived the change: the company reports a 0.98 rank correlation between model rankings on version 1.1 and on the original. AA-Omniscience tests what models know and whether they admit when they do not know. Its main metric, Net Score, gives credit for correct answers, subtracts for wrong ones and counts a declined question as neither, so a wrong guess costs a model points that declining to answer would not. Intel also has coverage of the benchmark's original launch.

The refresh removes questions that had been public and questions that had become saturated, meaning so many models answer them correctly that they no longer separate one model from another. It also corrects errors and ambiguous wording and adds new questions. The set still covers the same six domains and 42 topics: science, engineering and mathematics and software engineering get 600 questions each, and finance, health, humanities and social sciences, and law get 450 each. The new public set has 300 questions on Hugging Face, calibrated to match the difficulty of the private set that produces the official scores. At release, Artificial Analysis says public and private Net Scores correlate at 0.99 with a median difference of 0.0 points, so for the models tested at release, the public set ranked and scored them almost exactly as the private set did. The update ships as part of the company's Intelligence Index v5 release, according to its announcement thread.

On the new set, Google's Gemini 4 Carbon (high) leads with a Net Score of 54, ahead of Anthropic's Claude Opus 5.5 at 48 and Claude Fable 5.1 at 46, both run at maximum settings with fallback. Accuracy, the share of questions answered correctly, orders them differently: Claude Fable 5.1 is highest at 68% and Claude Opus 5.5 next at 66%. Because wrong answers cost points and abstentions do not, a model can lead on Net Score without leading on accuracy if it answers wrongly less often. Gemini 4 Argon (high) has the lowest hallucination rate, 4%, where hallucination rate measures how often a model answers a question it does not know. The best open-weights model, Muse Spark Open (xhigh), sits 28 points behind Gemini 4 Carbon.

For most models the refresh changes little. Net Scores fell 1.9 points on average and 0.6 points at the median, models answered a median 1.5 percentage points fewer questions correctly, and the median hallucination rate fell 1.4 percentage points. A few models moved much further: MiniMax M3.1 Flash dropped 22 points, from 16 to -6, and MiMo V2.6 Pro dropped 13, from 8 to -5. The thread does not say why those two fell so far. All of these figures come from Artificial Analysis running its own proprietary benchmark, and because scores shifted between versions, a v1 score and a v1.1 score should not be compared directly.