Scaling Laws, Carefully

Scaling laws help researchers estimate how to divide a training budget between model size and data. In Scaling Laws, Carefully, Lilian Weng explains how measurements from smaller training runs can be fitted to curves that predict loss, a measure of prediction error, at larger scales.

The post stresses that this is delicate because the fit is made on models cheap enough to train and then extrapolated to ones orders of magnitude larger. In that setup, choices that look like rounding error can lead to very different predictions: it points to reported exponents rounded to two digits and to rounded loss values changing fitted parameters. Repeated data in the training mix can also bend learning curves and make fitting less accurate. A scaling curve is therefore guidance conditioned on how the experiments and fit were done.

“The observation is simple in form: the training loss $L$ decreases predictably as we scale up model size $N$, dataset size $D$, and compute $C$, following a power-law curve, which appears as a straight line on a log-log plot.”