A super long overdue (3+ years?) post on scaling laws. Compute is expensive.

Scaling laws help researchers decide how to divide a training budget between a larger model and more training data. In the article linked by the post, Lilian Weng explains how measurements from smaller training runs are used to estimate prediction error at larger scales.

The estimate depends on the experiment behind it. Changing the training setup or repeatedly reusing limited data can alter the relationship being projected. The longer explanation therefore treats these curves as conditional planning tools: useful for comparing training allocations, with assumptions that must remain visible when extrapolating.