pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Why it matters
If you're fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → models and blocked by the cost of human annotation, this walks through concrete synthetic data generation strategies for instruction-tuning, preference-tuning, and pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → — a practical path around the annotation bottleneck.
Key quotes
“Furthermore, the quality and diversity of synthetic data often exceeds that of human annotators, leading to improved performance and generalization when models are finetuned on synthetic data.”
Eugene Yan
“While 92% of generated instructions were valid, synthetic input-output pairs were noisy. Overall, only 54% of the samples had completely valid fields.”
Eugene Yan
“I was astounded that synthetic data that was only half correct was useful for finetuning.”
Eugene Yan
“Results showed that Self-Instructed gpt-3 outperformed vanilla gpt-3 by 33% and nearly matched the performance of InstructGPT-001.”
Eugene Yan
“Alpaca finetuned llama-7b on 52k instruction-following samples generated from gpt-3.5 (text-davinci-003). They reported that this cost less than $500.”
