A practical explainer on what scaling laws predict and how to use them to allocate compute between data and model size before committing to an expensive training run — useful if you're planning or budgeting large model training.
A super long overdue (3+ years?) post on scaling laws.
Transcript
A super long overdue (3+ years?) post on scaling laws.
Compute is expensive. Scaling laws are a way to help us reason about the optimal compute allocation between data and model size before committing to a large run.
The post covers what scaling laws predict, how
