Vibeleaderboard
← All Intel
Intel / video

On the slow death of Scaling (birth of Adaption Labs) | Sara Hooker | HF ML Club India EP2

Source
youtube.com
Author
Hugging Face
Date

Context

Sara Hooker, a co-founder of Adaption Labs who previously led research at Cohere, argues in a Hugging Face ML Club talk that making models ever larger is yielding less. The first evidence is the history of the retired Open Leaderboard: models under 13 billion parameters improved steadily, and Hooker says small models now frequently outperform large ones, so size alone is not a predictable recipe.

Other evidence is an older paper finding that a small set of weights can predict about 95% of a network's weights, and that weights can be removed after training with minimal loss in performance. Hooker also says GPT-4.5, much larger, was widely seen as a modest gain and was replaced by routing because it was costly to serve. Hooker sees better returns now in post-training, and adaptive compute, and calls this a minority view. The examples are Hooker's selections, not a systematic survey.

Terms in this piece · Glossary
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
  • test-time compute — Spending more computation when the model answers — thinking longer, trying multiple attempts — to buy accuracy without training a bigger model.
Read the source www.youtube.com
More from Hugging Face
Recommended reads
Comments

Checking sign-in…

Loading comments…