Learning with not Enough Data Part 3: Data Generation

A language model can create labeled training examples by starting with a requested category and generating text that should fit it. Lilian Weng's data-generation survey explains how this can supply examples for training a separate classifier when human-labeled data is scarce.

The requested category is not proof that the generated text belongs to it. In the LAMBADA method the survey describes, a classifier checks whether its predicted label matches the requested one, and only the highest-confidence generated samples are kept, about the top 10 percent of ten times as many as needed. This addresses a central weakness of synthetic data: a learner can absorb mislabeled examples along with useful ones. Generation and quality control therefore form parts of the same training-data process.

“The goal of data augmentation is to modify the input format (e.g. text wording, visual appearance) while the semantic meaning stays unchanged.”

“Few shot prompting is shown to be effective for LM to learn within context without extra training.”