Measuring Healthcare AI Where It Actually Works

Vals built two healthcare benchmarks around jobs hospitals are already buying AI for, rather than around general medical knowledge. MedScribe asks a model to turn a doctor-patient conversation into a SOAP note, the standard structured clinical summary, and grades it against 100 expert-written rubrics across 80 transcripts. MedCode asks the model to assign ICD-10-CM diagnosis codes, the codes used for billing, across 2,755 samples. Both use de-identified patient records curated with the data company Protege, were annotated by certified professional coders, and hold out whole patients so a model cannot score well by having memorized them.

The best results differ by about 32 points. On MedScribe the leading models land between 85% and 88%, with GPT 5.1 at 88.09% and Claude Opus 4.6 at 86.74%. On MedCode most leading models sit between 49% and 53%, and the top score, from Gemini 3 Flash, is 55.92%. Vals' write-up explains the gap by the nature of each task. Writing a note means tracking a conversation, picking out what is clinically relevant and organizing it, which language models already handle. Coding means knowing which code takes precedence when a patient has several conditions, how comorbidities change the choice, and which of several near-identical codes fits, under rules Vals describes as hierarchical and unforgiving. Models did reasonably on physical conditions such as diabetes and hypertension and worse on mental health diagnoses, where the conventions require more layered judgment.

The two numbers come from different benchmarks with different grading, one rubric-based and one code-matching, so the gap is a comparison of task difficulty as Vals measured it rather than a single controlled experiment. The model list also reflects when the benchmarks were run. Vals' own reading is that AI can meaningfully assist with documentation now, while reliable automated coding still needs models that can apply precise rules at scale, and a wrong code carries reimbursement, compliance and audit consequences.