Explores why over-parameterized deep nets still generalize well
Covers the Lottery Ticket Hypothesis and generalization theory
For ML practitioners puzzled by deep learning overfitting paradox
Key quotes
“Since a typical deep neural network has so many parameters and training error can easily be perfect, it should surely suffer from substantial overfitting.”
Transcript
[Updated on 2019-05-27: add the section on Lottery Ticket Hypothesis.] If you are like me, entering into the field of deep learning with experience in traditional machine learning, you may often ponder over this question: Since a typical deep neural network has so many parameters and training error can easily be perfect, it should surely suffer from substantial overfitting. How could it be ever generalized to out-of-sample data points?
