A clear walkthrough of the main approaches for training when labeled data is scarce, giving practitioners concrete semi-supervised learning strategies to squeeze more value from unlabeled datasets.
Key quotes
“Active learning : Labeling is expensive, but we still want to collect more given a cost budget. Active learning learns to select most valuable unlabeled samples to be collected next and helps us act smartly with a limited budget.”
Lilian Weng
“Interestingly most existing literature on semi-supervised learning focuses on vision tasks. And instead pre-training + fine-tuning is a more common paradigm for language tasks.”
Lilian Weng
“Consistency Regularization , also known as Consistency Training , assumes that randomness within the neural network (e.g. with Dropout) or data augmentation transformations should not modify model predictions given the same input.”
Lilian Weng
“Temporal Ensembling keeps track of an EMA of label predictions for each training sample as a learning target. However, this label prediction only changes every epoch , making the approach clumsy when the training dataset is large.”
Lilian Weng
“Input augmentation (e.g. random flips of input images, Gaussian noise) or student model dropout is necessary for good performance. Dropout is not needed on the teacher model.”
