How to Generate and Use Synthetic Data for Finetuning

Synthetic training data becomes useful through selection as well as generation. Eugene Yan's survey uses Self-Instruct to explain one approach: start with human-written tasks, ask a model to generate additional instructions and answers, then filter the results before training on them.

The Self-Instruct paper describes removing unsuitable or overly similar examples and evaluating the trained model on separate tasks. This creates a way to expand instruction-training material from a small starting collection. It also makes two questions distinct: whether generated examples look usable, and whether training on them improves performance beyond those examples.

“Furthermore, the quality and diversity of synthetic data often exceeds that of human annotators, leading to improved performance and generalization when models are finetuned on synthetic data.”

Eugene Yan

“While 92% of generated instructions were valid, synthetic input-output pairs were noisy. Overall, only 54% of the samples had completely valid fields.”

Eugene Yan

“I was astounded that synthetic data that was only half correct was useful for finetuning.”

Eugene Yan

“Results showed that Self-Instructed gpt-3 outperformed vanilla gpt-3 by 33% and nearly matched the performance of InstructGPT-001.”

Eugene Yan

“Alpaca finetuned llama-7b on 52k instruction-following samples generated from gpt-3.5 (text-davinci-003). They reported that this cost less than $500.”

Eugene Yan