A model's confidence score becomes useful for decisions only when its relationship to actual correctness is understood. The essay The Normalization of Inexplicable Failures argues that teams also need to consider the cost of a wrong answer before choosing a threshold for action.
The concern is organizational as much as technical: accepting AI mistakes as inherently inexplicable can become an excuse to stop investigating failures. The essay argues for maintaining test cases and known answers so teams can check the system they are shipping. Its criticism concerns abandoning that responsibility, rather than treating a confidence number as sufficient evidence that an action is reliable.
Terms in this piece · Glossary
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
