⚡ Open Model Pretraining Masterclass — Elie Bakouch, HuggingFace SmolLM 3, FineWeb, FinePDF
Source
youtube.com
Author
Latent Space
Date
Why it matters
Covers how an open team trains small LMs and builds pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → datasets, practical knowledge if you fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition →, evaluate or deploy compact open models.
Terms in this piece · Glossary
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
videoThe State of Reasoning — from Nathan Lambert, Interconnects/AI2 [LS Live @ NeurIPS 2024]
video⚡️ARC-AGI-3: The Interactive Reasoning Benchmark
videoThe Agents Economy Backbone - with Emily Glassberg Sands, Head of Data & AI at Stripe
videoOne Year of MCP — with David Soria Parria and AAIF leads from OpenAI, Goose, Linux Foundation
