Do Agents Know When They Succeed? Calibrating Agent Confidence from Internal Representations

An agent's internal numerical states can help assess whether a completed run succeeded. This study trains confidence estimators using changes in those states across a run or representations associated with action decisions. It tests them on shell-command, database-query and programming tasks with programmatically checked outcomes, using models of roughly 7 to 14 billion parameters.

The reported results show useful success signals beyond the probabilities assigned to generated text, although performance varies by task and method. The researchers extracted internal states by replaying completed trajectories through the model. That experimental setup supports confidence scoring after a run; it does not establish a deployed monitor that warns before a failure occurs.

“we investigate whether model's internal representations provide stronger signals of eventual task success in multi-turn agentic setups”

“our methods consistently outperform surface level generation and sequence-based calibration baselines providing a zero-overhead reliability monitor that requires neither prompt alterations nor multi-sample rollouts”