LLMs answer drug questions from name patterns, not drug knowledge

Many drug names carry their class in a standardized ending; "-cillin," for instance, signals an antibiotic. The study behind this item, by researchers at UT Austin, Northeastern and MD Anderson and accepted to EMNLP 2026, shows that language models use that ending as a shortcut. Across ten models, an invented name such as "dimicillin" was routinely treated as an antibiotic and given plausible clinical details. The models tuned for medicine, Meditron3 and HuatuoGPT, were the most likely to accept invented drugs as real when asked open-ended questions.

To measure how much of a real drug's answer comes from the ending, the authors took each of 653 drug names and built versions with the unique part of the name replaced by a nonsense string, the ending replaced, or both, ten variants each. Comparing the model's answer probabilities across those versions shows whether it is drawing on the ending, the unique part, or knowledge tied to the whole name. For Ai2's Olmo 3 7B Instruct, 334 to 384 drugs, depending on the question format, produced behavior that could not be reliably told apart from a fully made-up name, and another 80 to 120 were driven mainly by the ending. Those counts are the 51–59% and 12–18% figures in Ai2's summary. Drugs with knowledge tied to the whole name got the most reliable answers.

Olmo was chosen because its training data and intermediate training checkpoints are public. Counting how often each drug appears in that data, the authors found that drugs the model saw rarely were more likely to be handled through the ending, and that the ending-based signal appeared earlier in training than drug-specific knowledge. They also traced the shortcut to the model's early-middle layers and found an internal direction there that can push the behavior up or down.

Two findings matter most for people asking models about medication. The models rarely mention the drug's name ending when explaining an answer that depends on it. And when 70 cases of models treating fictitious drugs as real were checked by hand, about 79% contained facts borrowed from real drugs, often by treating the invented name as another name for a real medication. Depending on the model, 66–98% of answers that accepted a fake drug expressed no uncertainty. The authors note that their method needs access to the model's answer probabilities, which rules out many closed models, and that the internal analysis covered only two 7B models.