AA-Omniscience v1.1 refreshes the hallucination benchmark
- Source
- x.com
- Date
We’ve released AA-Omniscience v1.1, updating our knowledge and reasoning benchmark with a refreshed public set, removal of saturated questions, and ambiguity fixes AA-Omniscience is our proprietary benchmark measuring model knowledge and hallucinations, including whether models admit when they don’t know an answer. Our primary metric, Net Score, rewards correct answers, penalizes incorrect answers, and treats abstentions neutrally. AA-Omniscience was updated to v1.1 as part of the Artificial Analysis Intelligence Index v5 release. Key changes in AA-Omniscience v1.1: ➤ Updated 3,000-question dataset: Down from 6,000 questions, with previously public and saturated questions removed, errors and ambiguities corrected, and new questions added ➤ Newly calibrated 300-question public set: AA-Omniscience v1.1 has a new public set on @huggingface calibrated to match the difficulty of the private set. At release, public and private scores have a 0.99 Pearson correlation and 0.99 Spearman rank correlation for AA-Omniscience Net Score, with a median difference of 0.0 points. Accuracy and hallucination rate also have 0.99 correlations, with median differences of -0.07 and -0.29 percentage…
AA-Omniscience v1.1 shrinks the dataset to 3,000 questions, adds a public 300-question set that tracks the private set at 0.99 correlation, and shifts scores (MiniMax M3.1 Flash drops 22 points). Re-check any model choices based on v1 numbers.
- AA-Omniscience's Net Score rewards correct answers, penalizes incorrect ones and treats abstentions as neutral, so a model that declines to answer when it does not know outscores one that guesses wrong.
- Version 1.1 cuts the set from 6,000 to 3,000 questions by removing previously public and saturated items, fixing errors and ambiguities, and adding new questions, while keeping the same 6 domains and 42 topics at 450 to 600 questions per domain.
- Declines were uneven: average Net Score fell 1.9 points against a median drop of 0.6, so the mean is pulled down by a few large falls, such as MiMo V2.6 Pro going from 8 to -5.
- Accuracy and leaders differ: Claude Fable 5.1 has the highest accuracy at 68% (Opus 5.5 at 66%), while Gemini 4 Argon has the lowest hallucination rate at 4%. The median hallucination rate fell 1.4 points versus v1.
- Gemini 4 Carbon (high) leads Net Score at 54, ahead of Claude Opus 5.5 at 48 and Claude Fable 5.1 at 46, while Muse Spark Open is the top open-weights model, 28 points behind the leader.
- hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
Checking sign-in…
Loading comments…