MLCR-AA ranks models on long medical case file reasoning
- Source
- x.com
- Date
Announcing MLCR-AA, our leaderboard for Wisedocs' MLCR (Medical Long Context Reasoning) benchmark for reasoning over long medical case files. We run the hardest, held-out question tiers, with Claude Fable 5 achieving the top score of 64.4% MLCR-AA tests models on realistic synthetic medical and insurance case files built by the team at @wisedocsai, based on the record review work their platform specializes in. Our leaderboard runs the private hold-out set of 60 questions from the two hardest categories: Expert, which requires specialist medical reasoning across a full case file, and Compound, which packs several independent questions into a single query. Each question is answered against a complete case file of ~70-150 pages, and graded by a three-model judge panel on completeness and accuracy alongside a concision test limiting verbose responses compared to expert answers. Accuracy verifies that the response is well-grounded in the source documents and case context while completeness assesses whether the model produced the same essential details that were included in expert-annotated responses. The concision test verifies that models are not producing excessively verbose…

Context
The most useful result in the MLCR-AA launch is that models were usually right about what they said but left out much of what an expert would include. Artificial Analysis built the leaderboard on Wisedocs' Medical Long Reasoning benchmark. Each of 60 held-back questions, drawn from its two hardest tiers, is asked against a complete synthetic medical and insurance case file of roughly 70 to 150 pages, the kind of review Wisedocs' platform does.
Three AI judges grade each answer on two separate axes. Accuracy checks that the answer is in the case documents; completeness checks whether it contains the essential details in an expert-written reference answer. A further test penalizes answers more than five times longer than the expert's, so a model cannot reach completeness by listing everything. Artificial Analysis reports that nearly 40% of models scored above 80% on accuracy while most scored below 50% on completeness. OpenAI's GPT-5.6 Terra had the highest accuracy, 93.7%, yet placed 10th overall.
At launch, Anthropic's Claude Fable 5 led with 64.4% at about $1 per task, and the median model scored under 15%. Artificial Analysis attributes Anthropic's lead to completeness, saying its models' accuracy was comparable to or below the strongest OpenAI models. These are document-review results on synthetic files, not measures of diagnosis. The live leaderboard has since added newer models and now headlines a pass rate, so the August figures are a launch snapshot rather than the current ranking.
- Context and context window — Context is the information available for a generation; the context window is the capacity available to hold it.
- Record — One unit of data in a file or table, such as one supplier's quote; distinct from a line of text.
- grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
- Open weights — A model whose learned parameter values have been made available to download, under terms the publisher sets.
Checking sign-in…
Loading comments…






