How a health-coaching agent tests conversations, not single replies

Numan, a UK digital health company, wrote on the LangChain blog about testing its AI health coach, Nu. The team says its standard evaluations failed because coaching is a conversation, not a single answer. Nu has to stay on the wellness side of a regulatory line. Under UK MHRA guidance, software that diagnoses, gives clinical advice or recommends treatment counts as a medical device. Numan started with the usual setup: fixed inputs, expected outputs, and an LLM judge scoring each reply. That setup misjudged the agent in both directions. Averaging scores across the whole dataset buried a few unsafe replies, so the system looked compliant when it wasn't. In one run, a rubric that expected empathy, exploration and advice within two turns marked Nu down for pacing a conversation the way a human coach would.

Scripting several turns in advance didn't fix it either. If the patient's second message is written before the test runs, it ignores what Nu actually said first. Once Nu goes off script, every later turn answers a conversation that never happened. Numan's answer is to keep fixed datasets only for what a single reply can show, such as routing, medication questions and whether the safety guard fires. Those datasets are scored per category, so a cluster of failures can't hide in an average. Safety probes gate releases automatically. For whole conversations, Numan is prototyping simulation: a cheaper model plays a patient with a persona, a scenario and a turn limit. It reacts to whatever Nu says, including adversarial attempts to slip a clinical request past the coach, and a judge then scores the full transcript. The authors say this kind of testing reduced serious scope violations in jailbreak scenarios. They don't give figures. Every eval runs through the same LangGraph agent that patients talk to, and Numan says the shared datasets turned a switch between frontier model providers from weeks of manual testing into a day.

The authors draw a lesson about who owns the test. Two rubric themes contradicted each other on missed doses: one said to refer the patient to a clinician, the other said to give general facts from the patient leaflet. That taught them that the evaluation criteria, not the prompt, are the scarce asset. On a regulated boundary, those criteria have to be written and signed off by a clinician. For that reason the simulated-conversation layer, which is still judged against engineer-drafted criteria, does not gate releases or back any safety claim yet.