← All IntelClip / AI AgentsOver-calibration is a failure mode, and perfect benchmarks are the wrong target
From Evals-Driven Development for a Mental Health AI Coach — Akele Reed & Dave Revere, SonderMind · ≈13:28
Three design choices — clinicians own the definition of good, labeled scenarios gate releases, and benchmarks are built from real failure modes — with an explicit warning that chasing benchmark scores drifts from real user need.
What’s in it
- Three design choices — clinicians own the definition of good, labeled scenarios gate releases, and benchmarks are built from real failure modes — with an explicit warning that chasing benchmark scores drifts from real user need.
Clip transcript
Right? So now we have a loop. And here's my next engineering problem for y'all. If we are truly designing a system with the human as the center node, then like AA said, that can't just mean that we trigger more, right? When my son is getting ready to move away and he's talking about packing up boxes, I don't want, you know, a system that's learned how to panic. I'll be doing the panicking. That might sound a little amusing, but the point is right that overc calibration can be a problem. It can prevent people from getting the care that they need. And so we've made three design choices around that calibration. The first is the clinical theme owns the definition of good. So vibes don't count here. An accountable judgment from a licensed expert does. And second, those labeled scenarios. So, we're asking concrete questions here. Did the expected observation fire? Uh, did the right category trigger? Did it happen at the right point in the conversation? Did the output evaluator catch the issue type? Okay. And so those labeled scenarios turn into evals that gate our releases. And here's our design philosophy around this one. We're not pursuing perfection with these benchmarks because that can actually cause us to drift our focus away from the human those benchmarks are supposed to protect, right? Because there can be real ambiguity in some of these edge cases. And so instead, our focus becomes how do we create benchmarks that serve real human needs by looking at real failure modes from real data. So false positives matter, false negatives matter, the category matters, the timing matters. We catch what matters and that's designing with the human as the center node.
Comments
Sign in to comment.
Loading comments…