Liquid AI d1-ultra tops the Artificial Analysis Decision Index
- Source
- x.com
- Date
Liquid AI has released d1-ultra, taking the #1 position on the Artificial Analysis Decision Index v1.0, supported by a reduction in hallucinations d1-ultra is the new decision model from @liquidai and follows on from their release of d1 earlier this month. It is priced at $0.042 per 1M input tokens compared to $0.02 per 1M input tokens for d1. Like d1, d1-ultra is billed only on input tokens. Key takeaways: ➤ d1-ultra leads the Artificial Analysis Decision Index v1.0 with a score of 65: d1-ultra scores 65, ahead of the recently released Jev 1.14 at 57 and GLiDE and Microsoft-Decision-1 at 54 ➤ d1-ultra’s score of 65 is driven by a much lower hallucination rate on AA-Omniscience-Decision: d1-ultra’s non-hallucination rate on AA-Omniscience-Decision is 85%, up substantially compared to d1 at 2%. The non-hallucination rate is the share of questions not answered correctly where the model chose "I don't know" and measures how well a decision model can judge its own knowledge in knowing when not to answer. While d1-ultra’s hallucination rate decreases, so does its accuracy (% questions correct). It achieves an accuracy of 32%, down 9 points compared to d1 (41%) ➤ d1-ultra has the third lowest Time per Task of any decision model: d1-ultra achieves a Time per Task of 92 ms, the third lowest of decision models measured, behind Jev 1.14 at 78 ms per task and Celeris-1 Decision at 63 ms per task ➤ d1-ultra offers comparable pricing to other decision models at $32 per 1M tasks: At $32 per 1M tasks, d1-ultra is comparably priced to Celeris-1 Decision ($32 per 1M tasks) and Microsoft-Decision-1 ($31 per 1M tasks), more than double the price of d1 and Mercury Decide at $15 per 1M tasks but less than half the price of GPT-6 Luna (Decisions) at $79 per 1M tasks
d1-ultra achieves the highest non-hallucination rate on AA-Omniscience-Decision at 85%, more than double the second-highest rate of 40% by Microsoft-Decision-1. Abstaining more compared to d1 leads to a lower accuracy of 32% compared to d1’s 41%.
d1-ultra leads When2Call at 93% accuracy, achieves equal first place with GLiDE on WildGuardTest with an average F1 of 95%, and equal second place with Decider V1.1 27B on HLE-MultiChoice with an accuracy of 15%.
d1-ultra achieves a time per task of 93 ms per task and at $32 per 1M tasks is comparably priced to Celeris-1 Decision ($32 per 1M tasks) and Microsoft-Decision-1 ($31 per 1M tasks).
Context
Decision models are a newer class of model that, instead of writing out an answer, pick from a fixed set of options and return a probability for each, all in one call, which Artificial Analysis says makes them cheap and fast and lets their confidence be checked for (artificialanalysis.ai). Its Decision Index, launched earlier this month (launch coverage), averages four equally weighted tests run by Artificial Analysis itself: multiple-choice reasoning questions from Humanity's Last Exam, NVIDIA's When2Call tool-calling test, WildGuardTest safety classification, and AA-Omniscience-Decision, a private knowledge test. That last quarter is split in half between accuracy and a non- rate, the share of questions the model did not get right on which it chose "I don't know" rather than a wrong answer.
That split explains how Liquid AI's new d1-ultra reached first place with 65 points, ahead of Jev 1.14 at 57 and GLiDE and Microsoft-Decision-1 at 54, even though it answers fewer knowledge questions correctly than its predecessor (Artificial Analysis thread). d1-ultra abstains far more often: its non-hallucination rate is 85%, against 2% for d1 and 40% for the next-best model, Microsoft-Decision-1, while its accuracy fell to 32% from d1's 41%. Under this index, declining a question it would have gotten wrong earns more than the lost correct answers cost. It also leads When2Call at 93%, ties GLiDE for first on WildGuardTest with an average F1 of 95%, and ties for second on the Humanity's Last Exam questions at 15% accuracy.
The higher score comes at roughly twice d1's price: Liquid charges $0.042 per million input versus $0.02 for d1, and like d1 it is billed only on input tokens. Artificial Analysis puts that at $32 per million tasks, in line with Celeris-1 Decision and Microsoft-Decision-1. Liquid's own launch post, which frames d1-ultra around calibrated probabilities on harder tasks, is covered separately; the ranking here is Artificial Analysis's independent measurement on a v1.0 index that it says it is still developing.
- calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
- hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
Checking sign-in…
Loading comments…