Tools Fail Detecting Silent Errors In Faulty Tools 2024 06 27

A tool can return a wrong answer without reporting an error. The Tools Fail study, listed by Cohere, examines this problem using a deliberately faulty calculator and tools involved in simulated household tasks. Correctly calling a tool is therefore only one part of reliable agent behavior. The publisher abstract identifies detecting silent failures as a separate challenge.

In the paper's calculator experiments, models often detected incorrect results more successfully when asked specifically to accept or reject them than when required to detect and correct them together. Some interventions also increased rejection of correct answers, showing why a checking step must be evaluated for false alarms as well as missed errors.