Caliber Calibrating Confidence Before And After Reasoning In Language Models 2026 06 24
Source
cohere.com
Author
Cohere
Date
Context
A model's confidence before solving a problem answers a different question from its confidence after producing an answer. Cohere's CALIBER research trains both estimates: the first predicts success on the problem, while the second assesses the particular answer produced. The authors report a 52.5% reduction in expected calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition → error against their strongest single-confidence baseline on BigMathDigits using a seven-billion-parameter model. That measures how closely confidence matches observed correctness. It supports training confidence against the information available at each stage; it does not mean answer accuracy increased by 52.5%. Publisher abstract.
Terms in this piece · Glossary
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
postCohere releases open-weights translation model North Small Translate
articleThe Data Provenance Initiative A Large Scale Audit Of Dataset Licensing And Attribution In Ai 2023 10 25
articleLanguage Models Don T Always Say What They Think Unfaithful Explanations In Chain Of Thought Prompting 2023 05 07Cohere
articleProcedural Knowledge In Pretraining Drives Reasoning In Large Language Models 2024 11 20Cohere
articleOptimismBench: Forecasting Bias and the Alignment Effect in Language Model JudgmentSeonglae Cho, Adriano Koshiyama