Repeated Pretraining Data Can Waste a Third of Compute
- Source
- x.com
- Date
Deduplication is standard practice but never perfect --this work measures what the residue costs in compute-equivalent terms, and shows the worst-case repetition structure is predictable from model size. The wrong combination can waste as much as 33% of compute! https://t.co/BbsOWtX0dS
Context
Deduplication tools rarely catch every repeated passage in a training corpus. Stanford SAIL's post points to work translating the resulting degraded performance into an equivalent amount of wasted compute, rather than just a loss number, in a paper by Jessica Chudnovsky, Joshua Kazdan, and coauthors that received an ICML 2026 workshop oral.
The authors report that repeating a moderately sized subset of data a moderate number of times damages performance more than either repeating a large subset rarely or a small subset very often, and that this worst-case combination is predictable given a model's size, wasting up to 33 percent of compute in that scenario.
- pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
articleScaling Laws And Interpretability Of Learning From Repeated DataAnthropic research
articleTraining Compute-Optimal Large Language ModelsJordan Hoffmann et al.
articlePretraining and adapting a language model on a dependency-free stack: GPT-2 124M from random weights, reproduced against llm.c, and a clinical adapter for Qwen3-0.6BThang Tran (CloudKites AI Lab, New South Wales, Australia), Lan Dang (Monash Business School, Monash University, Victoria, Australia)
Checking sign-in…
Loading comments…