Scaling Laws, Carefully
lilianweng.github.io- Category
- Other
- Type
- ARTICLE
- Builder
- @lilianweng
- Added
- Jul 21, 2026
About
Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss $L$ decreases predictably as we scale up model size $N$, dataset size $D$, and compute $C$, following a power-law curve, which appears as a straight line on a log-log plot. We can view scaling laws as a framework for describing the relationship between compute, loss, model size and data; at its core, it is about how to allocate precious compute optimally between $N$
Why it made the leaderboard
Walks through how training loss scales predictably with model size, data, and compute, and how to allocate a fixed compute budget optimally between parameters and tokens — essential grounding for anyone planning or interpreting large training runs.
Comments (0)
No comments yet
Indexed by a proprietary survey. Corrections welcome.