When a model reads a drug's class from its name—not its knowledge
Source
allenai.org
Date
Context
A plausible answer about a medication can come from recognizing its name's ending. A study described by Ai2 tested Olmo 3 7B Instruct by replacing drug-name stems, affixes or both with invented strings. Comparing answers to real and altered names helped distinguish name-based shortcuts from drug-specific knowledge. The research account describes the diagnostic.
Depending on the evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → format, 12–18% of tested drugs showed affix-driven behavior. Searches of the open training corpus also linked rarer drug names with greater reliance on affixes. The result identifies a specific weakness in this model's answers: recognizing a drug family can substitute for knowing the individual medication.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
