A smaller classifier recovered most of a larger model's accuracy when uncertain cases were escalated. In OpenRouter's test of 3,080 banking messages, TypeSafe's Jev scored 81.0% accuracy and Anthropic's Claude Opus 5 scored 84.4%. Sending cases below Jev's 0.90 confidence threshold to Opus produced 84.0% accuracy at $0.69 per thousand messages, versus $2.42 for Opus alone, a figure that already reflects cached prompts with reasoning turned off. The experiment supports selective escalation on this dataset. The threshold was assessed on those same examples, and Jev's confidence scores were not calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition → probabilities.
Terms in this piece · Glossary
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
