Stanford Researchers Test Pretraining a Model on Self-Generated Data Alone

Language models are normally pretrained on huge amounts of real text scraped from the internet. Researchers including Stanford AI Lab's Aditya Cowsik and Kfir Dolev asked what happens if a randomly initialized model instead generates all of its own training data through self-play, with no real text involved at any point, in a paper titled Self-Play Pretraining with Zero Data.

According to the paper's abstract, this is an early proof-of-concept: the model learns by playing a kind of generation game against itself, and the researchers then test it on real text it was never shown. They report that as they increase the amount of self-play compute, the model's zero-shot performance on that real text improves in a predictable, scaling-law-like pattern, and that the model also develops in-context learning purely from this synthetic self-play. The authors describe this as a demonstration that the approach works at all, not a substitute for pretraining on real data at current scale.