Are Deep Neural Networks Dramatically Overfitted?

A neural network can fit its training examples perfectly and still perform well on unfamiliar examples. That makes parameter count and training accuracy insufficient tests of whether it has merely memorized its data. Lilian Weng's 2019 survey explains this puzzle by contrasting generalization with the ability to fit random labels.

In the underlying experiments, image classifiers could also learn labels that researchers had randomly reassigned. Fitting those labels shows capacity to memorize, without establishing useful predictive structure. The distinction is between reproducing training answers and succeeding on held-out examples; the former alone cannot establish the latter.

“Since a typical deep neural network has so many parameters and training error can easily be perfect, it should surely suffer from substantial overfitting.”