Do Agents Know When They Succeed? Calibrating Agent Confidence from Internal Representations
Source
Priyanka Mary Mammen, Emil Joswin, Srujananjali Medicherla
Author
Priyanka Mary Mammen, Emil Joswin, Srujananjali Medicherla
Date
Terms in this piece · Glossary
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
Probing an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →'s internal residual-stream activations for success prediction outperforms surface and sequence-based calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition →, offering a near-zero-overhead reliability signal usable in production agent pipelines.
Key quotes
“we investigate whether model's internal representations provide stronger signals of eventual task success in multi-turn agentic setups”
“our methods consistently outperform surface level generation and sequence-based calibration baselines providing a zero-overhead reliability monitor that requires neither prompt alterations nor multi-sample rollouts”
articleA Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost after Coding-Agent AdoptionYegor Denisov-Blanch, Shyam Agarwal, Pavel Azaletskiy, Hao He, Rylan Schaeffer, Brando Miranda, Bogdan Vasilescu, Sanmi Koyejo
articleEngineering Reliable Coding Agents: Evaluating and Operating the System Around the ModelStephanie Jarmak
articlePosition: Behavioral Systems Require Behavioral TestsManuel Cherep, Nikhil Singh, Pattie Maes