Stanford CS25: Transformers United V6 I From Next-Token Prediction to Next-Generation Intelligence
- Source
- youtube.com
- Author
- Stanford Online
- Date
Context
In a Stanford CS25 lecture, Shrimai, an AI scientist who had recently joined Mistral AI after working on NVIDIA's Nemotron models, asks whether the order a sees its data matters. Her two-phase recipe first trains on a broad, diverse mix including medium and low-quality web text, then switches to higher-quality sources such as math, code and Wikipedia, repeated more times. Quality classifiers set the mix weights, and ablation tests decide how many repeats help before gains shrink.
She reports that this model averaged 17 percent better than a baseline sampled in proportion to each source's size, and 3.4 percent better than the same quality-weighted blend shown in random order. Swapping the phases, quality first, gave weaker results. These are averages across downstream tasks from her team's NVIDIA work, as presented in the talk.
- Model — A computational system that uses learned patterns to produce an output from information supplied to it.
- Training — The process that adjusts a model using examples and a measure of how well it performs.
- pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
videoStanford Webinar - Building Trustworthy AI: Navigating Security in the Agentic Era
videoStanford CS547 HCI Seminar | Autumn 2025 | HCI Systems in the Age of AI Code Generation
videoStanford CS25: Transformers United V6 I Advancing Science and Medicine with Collaborative AI Agents
videoStanford CS153 Frontier Systems | Andreas Blattmann from Black Forest Labs on Visual Intelligence
videoStanford CS25: Transformers United V6 I From Language Models to Native Multimodal IntelligenceStanford Online
videoStanford CS25: Transformers United V6 I On the Tradeoffs of State Space Models and TransformersStanford Online
videoBuilding the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan AnilSequoia Capital
Checking sign-in…
Loading comments…