Clip / AI Agents

They had to turn off the frontier providers' built-in guardrails on day one

From Evals-Driven Development for a Mental Health AI Coach — Akele Reed & Dave Revere, SonderMind · ≈19:55

Direct answer to a practitioner question: stock provider safety layers filtered essentially every mental-health test case, so domain-correct guardrails had to be built from scratch and the false-positive margin narrowed deliberately.

What’s in it

Direct answer to a practitioner question: stock provider safety layers filtered essentially every mental-health test case, so domain-correct guardrails had to be built from scratch and the false-positive margin narrowed deliberately.

Watch this moment

Clip transcript

me? >> Yeah, >> there we go. Okay. Um well, uh first question. Um, so yeah, we like day one we had to turn off the like uh built-in guardrails because general purpose LLMs are overc calibrated and so we we built our own um our own guardrails as a result. Uh yes, we had to turn off those those ones because you're exactly right like we would try to run our data sets and it would just like filter everything. Um and then uh the second question uh similarly we like overc calibration is a compassionate choice from both the frontier model uh providers and also on our side um we try to make that margin obviously much smaller right um so that again they're more correct uh but yeah the over overc calibration so that's the I guess that's the short answer