token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
Why it matters
Walks through how training loss scales predictably with model size, data, and compute, and how to allocate a fixed compute budget optimally between parameters and tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → — essential groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → for anyone planning or interpreting large training runs.
Key quotes
“The observation is simple in form: the training loss $L$ decreases predictably as we scale up model size $N$, dataset size $D$, and compute $C$, following a power-law curve, which appears as a straight line on a log-log plot.”
