The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside
Source
AI Engineer
Author
AI Engineer
Date
Terms in this piece · Glossary
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
Why it matters
A rare first-party account of the failure modes inside a large pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → run, including the bugs that silently flatten a loss curve and the replica-equality check that catches them.
Key quotes
“the rule of thumb is if task is too hard for your model, then your model will start to fall on its face. Lose correctness, lose diversity.”
Marah Abdin
“synthetic data gives us a track to extract some of these features and project them on some new planes”
Marah Abdin
“If you've got data that sucks, you can't train a good model. If you've got a training code base that sucks, you also can't.”
Robert McHardy
“we noticed about 0.5% of the gradient gets silently corrupted, essentially replaced by random values”
Robert McHardy
“The recipe held, it scaled, and we will continue scaling it from here.”
