A probability attached to an answer is useful only if it tracks how often the answer is right. Artificial Analysis' Decision Index evaluates models that choose among fixed options and return probabilities, making their confidence measurable alongside speed, cost and accuracy.
The launch results found widespread overconfidence, especially on knowledge and reasoning questions, with better calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition → on classification and tool-choice tasks. A system using confidence to route work or request human review therefore needs to check calibration on its own task type. Fast structured decisions and trustworthy confidence are separate properties; a clean probability field does not by itself supply the latter.
Terms in this piece · Glossary
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
tool use — A model's ability to call external functions — run code, search the web, edit files — instead of only generating text.
