If you're training models with scarce labels and a tight annotation budget, this walks through active learning methods for choosing which samples are worth labeling — a practical lever for maximizing model quality per dollar of human labeling.
Key quotes
“This is part 2 of what to do when facing a limited amount of labeled data for supervised learning tasks. This time we will get some amount of human labeling work involved, but within a budget limit, and therefore we need to be smart when selecting which samples to label.”
Lilian Weng
“deep learning model predictions are often not calibrated and not correlated with true uncertainty well. In fact, deep learning models are often overconfident.”
Lilian Weng
“Unfortunately all the cheap implicit ensemble options above perform worse than naive ensembles. Considering the limit on computational resources, MC dropout is still a pretty good and economical choice.”
Lilian Weng
“When the number of classes grows to be large or the data dimensionality increases (“curse of dimensionality”), the core-set method becomes less effective ( Sinha et al. 2019 ).”
Lilian Weng
“Coleman et al. (2020) experimented with a weaker model (e.g. smaller, weaker architecture, not fully trained) and found that empirically using a weaker model as a proxy can significantly shorten each repeated data selection cycle of training models and selecting samples, without hurting the final error much.”
