On the slow death of Scaling (birth of Adaption Labs) | Sara Hooker | HF ML Club India EP2
- Source
- youtube.com
- Author
- Hugging Face
- Date
Context
Sara Hooker, a co-founder of Adaption Labs who previously led research at Cohere, argues in a Hugging Face ML Club talk that making models ever larger is yielding less. The first evidence is the history of the retired Open Leaderboard: models under 13 billion parameters improved steadily, and Hooker says small models now frequently outperform large ones, so size alone is not a predictable recipe.
Other evidence is an older paper finding that a small set of weights can predict about 95% of a network's weights, and that weights can be removed after training with minimal loss in performance. Hooker also says GPT-4.5, much larger, was widely seen as a modest gain and was replaced by routing because it was costly to serve. Hooker sees better returns now in post-training, and adaptive compute, and calls this a minority view. The examples are Hooker's selections, not a systematic survey.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- test-time compute — Spending more computation when the model answers — thinking longer, trying multiple attempts — to buy accuracy without training a bigger model.
Checking sign-in…
Loading comments…






