Artificial Analysis launches Decision Index for classifier-style models
- Source
- x.com
- Date
Introducing the Artificial Analysis Decision Index, benchmarking decision models across classification, tool calling, knowledge and reasoning. Jev leads among decision models; language models such as Claude Haiku 5.5 still deliver higher accuracy, but at much higher cost and latency Decision models deliver structured answers or classifications with associated probabilities, offering faster responses and lower costs than using language models. The Artificial Analysis Decision Index v1.0 compares the capability, cost, and speed of decision models and also compares to language models. Benchmarks in the Artificial Analysis Decision Index v1.0: ➤ AA-Omniscience-Decision: A private dataset of 500 questions adapted from the proprietary AA-Omniscience benchmark, measuring decision models’ knowledge and tendency to hallucinate ➤ HLE-MultiChoice: 513 text-only multiple-choice questions from Humanity’s Last Exam, testing decision models’ reasoning capabilities ➤ When2Call: A tool-calling benchmark from NVIDIA, testing whether models can correctly decide to call a tool, request clarification, decline to answer, or respond directly, evaluating capabilities relevant to agentic workflows…
Context
Decision models return a selected answer or classification with an associated probability. Artificial Analysis's Decision Index announcement reports that leading examples can match language models on classification and tool-routing tasks while responding faster and costing less. The same models trail more substantially on knowledge and reasoning tasks, so the advantage depends on the kind of decision being made.
Their probabilities also need interpretation. The evaluator reports that on knowledge and reasoning questions, actual accuracy commonly falls below the probability assigned to the chosen answer. A confident output therefore cannot automatically be treated as an equally reliable estimate of correctness.
Checking sign-in…
Loading comments…