Learning with not Enough Data Part 1: Semi-Supervised Learning

Unlabeled images can help train a classifier alongside examples already labeled by people. In a semi-supervised learning survey, Lilian Weng explains FixMatch: the model predicts a label for a lightly altered image, keeps sufficiently confident predictions, then learns to produce the same label for a more heavily altered version.

Those predictions become temporary training targets, often called pseudo-labels. Matching them across transformations encourages the classifier to recognize the same content despite changes in appearance. Confidence filtering limits which predictions become targets, although it cannot make an incorrect prediction true. The central opportunity is to extract additional training signals from existing unlabeled data.

“I do feel the assumption that the class distributions on the labeled and unlabeled data should match is too strong and not necessarily to be true in the real-world setting.”

Lilian Weng

“Noise is important for the student to perform better than the teacher.”