← All IntelClip / AI ToolsSynthetic data as a complement, at 13% of the pre-training mix
From The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside · ≈1:57
A rare concrete number for how much synthetic data a frontier-adjacent lab actually puts in pre-training, plus the rationale: surfacing rationale, planning and structure that organic data leaves implicit.
What’s in it
- A rare concrete number for how much synthetic data a frontier-adjacent lab actually puts in pre-training, plus the rationale: surfacing rationale, planning and structure that organic data leaves implicit.
Clip transcript
relied a lot more on synthetic data in a few forms which I'm going to. Okay, so before we kind of going into what does that mean and what have we done, etc. One uh why would we kind of It's It's Sometimes it's fair at least to ask why synthetic data and the thing is that uh at least at Pulsar we don't see it as a way to replace organic data. I don't see it so in the current state of the world at least, but uh it is a way to kind of complement it. And the thing is that organic data has a lot in it that is basically kind of implicitly hidden. A lot of things that could teach the model or not very presented in the most optimal way sometimes. And so synthetic data gives us a track to extract some of these features and project them on some new planes. And this is how we get to expose um implicit rationale, implicit planning, implicit structure, and a way for us to fill gaps and regularize not only how we present the tokens, but also how we are teaching the model. Um for access point two in particular, we settled on 13% of the mix. This is only pre-training stages before post-training. Um and since then we've just been continuously generating more data in a bunch of directions. Now we have a six trillion token uh corpus that's continuously growing. Okay.
Comments
Sign in to comment.
Loading comments…