Will We Run Out Of Data Limits Of Llm Scaling Based On Human Generated Data

This Epoch AI article, published June 6, 2024, summarizes a paper by Pablo Villalobos and colleagues that updates Epoch's 2022 estimate of how much public human-written text exists for training language models (epoch.ai). Its central estimate is an effective stock of about 300 trillion tokens, with a 90% confidence interval of 100 to 1,000 trillion. That counts only text good enough to train on, and it allows for training on the same data more than once. Combining two models of how fast datasets grow, the authors put 80% probability on this stock being fully used sometime between 2026 and 2032.

The date depends on a choice developers make. Compute-optimal training balances model size against data for the best performance at a given compute budget. Under that approach, there is enough text for a training run of about 5e28 FLOP, which the paper expected in 2028. Many recent models are instead overtrained: smaller models fed more data, which makes them cheaper to run. Llama 3-70B was overtrained by 10x. If models are overtrained by 5x, the stock is used up by 2027, and at 100x it would have been used up by 2025. The 2022 paper had predicted exhaustion by 2024. The authors attribute the later date to two findings: carefully filtered web data can outperform curated corpora, which raised their estimate of high-quality data about fivefold, and models can train on several passes over the same data without much loss, which added another 2-5x.

The paper does not treat exhaustion as a hard stop. Growing models while keeping data fixed can deliver the equivalent of up to two more orders of magnitude of compute-optimal scaling before progress plateaus. Beyond 2030 it expects progress to require new techniques: synthetic data, learning from other kinds of data, and better data efficiency. At the time, it described synthetic data as reliably helpful only in narrow domains such as math and coding. Epoch's September 2025 report What will AI look like in 2030? later called a data bottleneck surmountable. It said public human text suffices to scale to at least 2027 and that synthetic data's usefulness is better established since reasoning models arrived.