The index now scores both correctness and calibrated confidence, and 45% of it is private data, which limits benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → gaming. Cost per 1M tasks varies over 60x across models, partly because of how document input is billed.
Key takeaways · AI-distilled
Artificial Analysis's Decision Index v2 scores each answer as 50 × (1 + c − (p − c)²): correct answers land above 50, wrong ones below, and confidence moves the score within each half. Over a set it equals 50 × (1 + accuracy − Brier score).
calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition → changes rankings. On AA-Paperclip, Clef passes 36% of documents to Celeris-1.1 Decision's 24%, yet they tie on Decision Score because Celeris has a Brier score of 0.15 against Clef's 0.27.
AA-Paperclip passes a document only if all 15 answers are right. Drex Max v1.1 gets 97% of individual answers right but passes 63% of documents; Claude Haiku 5.5 (Max) gets 99% and passes 92%.
Billing method drives cost: GLiDE v1.2 bills a document's input for every question, about 63k input tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → per AA-Paperclip task against about 9k for Drex Max v1.1, so it costs $780 per 1M tasks, nearly 8x Drex Max, at only 33% higher list prices.
General language models score higher at a steep price: per Artificial Analysis, GPT-6 Luna (Max) scores 76 to Drex Max v1.1's 73 but costs 15x more ($1,530 vs $102 per 1M tasks) and is about 40x slower (11.34 s vs 275 ms per task).
Terms in this piece · Glossary
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
articleVials Aa
articleThinking Machines Releases Inkling 2 Small Highest Scoring Open Weights Model Outside Of Chinia Upon Weights Release It
articleNanbeige Has Released Nanbeige4 3 33b Scoring 20 On The Intelligence Index
articleGithub Copilot Fusion Orchestrates Anthropic And Openai Models To Balance Performance And Cost
