Don't Mock Machine Learning Models In Unit Tests

A test can pass while software misinterprets a model's prediction. Eugene Yan's testing essay describes the problem: replacing a model with a fabricated output can hide mistakes in how the application maps output positions to class meanings.

The essay recommends keeping fast tests for ordinary processing logic while reserving selected checks for the actual model. Randomly initialized models can test structural properties such as output dimensions, but learned behavior requires learned weights. Small real-model checks can therefore catch a different category of failure from a mock, without requiring every unit test to load a large model.

“In software, we typically mock dependencies like APIs; in ML, we want to test the actual model (sometimes).”

Eugene Yan

“Google’s T5 NLI model classifies factual consistency with class = 1 while Meta’s BART NLI model classifies it with class = 2!”

Eugene Yan

“Don't Mock Machine Learning Models In Unit Tests”

Eugene Yan