Training Compute-Optimal Large Language Models

It reset how everyone spends a training budget. That a smaller model trained on far more data beats a larger undertrained one is why model sizes stopped climbing while token counts exploded — and why a parameter count on its own now tells you almost nothing.

• token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

• postWhy parameter count alone can't tell you if a model is well-scaledjietang
• articleScaling Laws for Neural Language ModelsJared Kaplan et al.
• articleMaking LLMs faster without sacrificing accuracywww.amazon.science

Checking sign-in…

Loading comments…