Intel / article

Healthcare Blind Spots: AI Models Prone To Fabricating Diagnoses

Source: ri.cmu.edu
Author: Mallory Lindahl
Date: Published Jul 20, 2026

From the Daily Brief

Researchers at Carnegie Mellon's Robotics Institute tested Claude, GPT-5, and Gemini by asking each model to diagnose a patient from an image that was never actually supplied. In 18% of these cases, the models fabricated a diagnosis anyway, inventing plausible sounding conditions grounded only in the patient's stated age, race, and gender rather than any real clinical evidence.

The failure mode matters because it does not look like an obvious hallucination. A fabricated diagnosis tied to demographic details reads as clinically reasonable to a non expert, and the study's framing suggests the models filled the evidence gap with statistical priors about which conditions correlate with which demographics, rather than declining to answer or flagging the missing image.

For anyone building or evaluating healthcare facing AI tools, the finding argues for testing explicitly for missing evidence handling, not just diagnostic accuracy on complete cases. A model that performs well on standard benchmarks can still confidently invent findings when the input it needs was never there, and demographic based confabulation raises its own bias concerns on top of the factual error.

Read the 2026-09-22 Brief →

Context

A missing medical image did not consistently stop tested AI models from inventing a diagnosis. A CMU Robotics Institute study report describes nearly 11,700 responses to image questions with the images deliberately omitted. Across the tested models, about 82% declined to answer, while 18% supplied fabricated diagnoses.

The prompts used simulated patient profiles, and the report says demographic information influenced the invented answers. This tests whether a system recognizes missing evidence. The reported percentage is not an error rate for diagnosing real patients from actual medical images.

Key takeaways · AI-distilled

In a CMU study, Claude, GPT-5, and Gemini fabricated a medical diagnosis in 18% of cases when the supporting image was deliberately left out of the question, rather than asking for the missing scan (about 11,700 model responses analyzed).
A simulated 65-year-old white man asking about a skin mole received a melanoma diagnosis in nearly every response, even though fewer than 1 in 10,000 moles actually become melanoma, per Memorial Sloan Kettering.
GPT-5 named sarcoidosis for roughly 77% of simulated young Black patients asking about a chest X-ray, a disease the Cleveland Clinic says affects fewer than 200,000 people in the U.S. at any given time.
Models correctly declined to answer without the image 82% of the time; the fabricated diagnoses in the other 18% were driven by the patient's stated age, gender, and race rather than any actual scan.
Researcher Siddharth Vohra tested chest X-ray, brain MRI, and dermatology scenarios across 12 simulated patient profiles, concluding that demographic-sensitivity audits are needed before such models are used in clinical settings.

Read the source www.ri.cmu.edu