If you write tests for ML pipelines, this explains why mocking models hides real failures and what to validate instead — practical guidance for keeping AI code reliable as it evolves.
Key quotes
“In software, we write code that contains logic; in ML, we write code that learns logic and then uses that learned logic.”
Eugene Yan
“In software, we typically mock dependencies like APIs; in ML, we want to test the actual model (sometimes).”
Eugene Yan
“For instance, two models may have different output classes: Google’s T5 NLI model classifies factual consistency with class = 1 while Meta’s BART NLI model classifies it with class = 2!”
Eugene Yan
“Don’t test external libraries. We can assume that external libraries work. Thus, no need to test data loaders, tokenizers, optimizers, etc.”
Eugene Yan
“Probablistic tests create an impossible problem: if you tighten your assertions you struggle with meaningless failing tests and normalize ignoring test failures, while if you loosen your assertions your tests aren’t really asserting anything any more.”
