Artificial Analysis rebuilds its medical long-context leaderboard
- Source
- x.com
- Date
We've updated MLCR-AA, our leaderboard for Wisedocs' Medical Long Context Reasoning benchmark, with Claude Sonnet 5.5 and GPT-6 Astra now tied for the #1 score We've updated how MLCR-AA prompts and grades models on @wisedocsai's benchmark, to better reflect how medical record review is done in practice. Models are now prompted as medical record reviewers, and asked to answer each part of a question in one paragraph with their conclusion and the supporting evidence from the record. Answers are graded by a new three-model judge panel, and we reviewed all 60 expert reference answers against their source records, refining 15. Claude Sonnet 5.5 (max with fallback) from @AnthropicAI and GPT-6 Astra (max) from @OpenAI are tied at the top with 41.7%, followed by GPT-6.1 Sol (max) at 38.3%. Qwen3.8 2.4T A95B from @Alibaba_Qwen is the leading open weights model at 30.6%. Key results from the updated MLCR-AA: ➤ Medical record review remains far from solved: an answer passes only when it is both complete and accurate, and no model passes more than 42% of questions and the median model passes under 15% ➤ Anthropic models lead on completeness, reaching the expert's conclusion with the…
No model passes more than 42% of long medical-record questions. Claude models lead completeness while GPT models lead accuracy, so model choice depends on which failure matters. Cost per task spans roughly $0.006 to $1.30.
- Artificial Analysis changed MLCR-AA grading: models now answer as medical record reviewers with a conclusion plus supporting record evidence, a three-model judge panel grades them, and 15 of 60 expert reference answers were refined after review.
- An answer passes only if it is both complete and accurate. Claude Sonnet 5.5 (max with fallback) and GPT-6 Astra (max) tie at 41.7%, GPT-6.1 Sol (max) follows at 38.3%, and the median model passes under 15%.
- Claude models hold the top 18 completeness scores, led by Sonnet 5.5 at 64.8% vs 42.2% for the best non-Anthropic model. OpenAI models hold the top 27 accuracy scores, led by GPT-6.1 Sol at 96.7%, which is complete on only 40.0% of questions.
- Qwen3.8 2.4T A95B is the leading open-weights model at 30.6%, about 11 points behind the two models tied for first.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Checking sign-in…
Loading comments…