Vibeleaderboard
← All Intel
Intel / article

Scaling Laws for Neural Language Models

Source
arxiv.org
Author
Jared Kaplan et al.
Date

Context

OpenAI's 2020 scaling-law study found regular relationships between training resources and text-prediction performance. Its metric, cross-entropy loss, measures how well a model predicts text; the researchers evaluated it on held-out data. Across the tested ranges, loss followed power-law trends with model size, dataset size and training computation. The paper used those empirical relationships to estimate how to allocate a fixed budget, favoring larger models stopped before training fully converged. The finding made resource allocation more predictable within the studied setting. It describes a text-prediction metric, rather than a universal law of intelligence.

Recommended reads
Comments

Checking sign-in…

Loading comments…