The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside

Synthetic training data can help when a model is seeing the same high-quality material too often. In Poolside's training talk, the team describes using generated rephrasings to replace some repeated material and expose reasoning or structure that source documents leave implicit.

The speakers distinguish inexpensive transformations closely tied to source material from more elaborate generation pipelines. For difficult tasks, they break generation into smaller steps and use filtering or validation components. Their account treats synthetic data as a complement to organic data: its purpose is to change what the model can learn from the available material, rather than merely increase the volume of text.

“the rule of thumb is if task is too hard for your model, then your model will start to fall on its face. Lose correctness, lose diversity.”

Marah Abdin

“synthetic data gives us a track to extract some of these features and project them on some new planes”

Marah Abdin

“If you've got data that sucks, you can't train a good model. If you've got a training code base that sucks, you also can't.”

Robert McHardy

“we noticed about 0.5% of the gradient gets silently corrupted, essentially replaced by random values”

Robert McHardy

“The recipe held, it scaled, and we will continue scaling it from here.”

Robert McHardy