← All IntelClip / AI ToolsMAI Thinking 1: no synthetic data, but web text still collapses to 15%
From The Base Model Is Dead — Varun Singh, Arcee AI · ≈6:18
“web text, which used to make up like up to 85% of the train data in GPT uh 3, is now all the way down at 15%”
What’s in it
- Contrasts two philosophies on synthetic vs human-only training data
- Breaks down how web text share collapsed from 85% to 15%
- Explains why code and STEM data now dominate LLM pretraining mixes
Clip transcript
got like two contrasting uh perspectives here kind of in the slide. The top image is from the MEI uh Thinking 1 paper where they make a make it really large point to not use any synthetic data or any uh data from any other language model. Um and they really try to, you know, filter their web scripts for this as well. Um in order uh to kind of adhere to like the previous um paradigm of like uh using human knowledge as a way to bootstrap uh model representations and like uh capabilities. Um But I would say that this is also, like even even though they stuck with no synthetic data, the data mix that they've chosen here is still um totally different from what you'd expect in like a um in a in a classical uh language model. And I mean, the the main reason for that is that web text, which used to make up like up to 85% of the train data in GPT uh 3, is now all the way down at 15%. And that I mean, that just shows that uh the value of like web text contributing to like the downstream um performance of like the models on RL and stuff is kind of uh it it's still important, but taking a backseat to things like code and stem abilities as the models kind of gain more real-world use cases related to to those.
Comments
Sign in to comment.
Loading comments…