Vibeleaderboard
← All Intel
Intel / video

Stanford CS25: Transformers United V6 I From Next-Token Prediction to Next-Generation Intelligence

Source
youtube.com
Author
Stanford Online
Date

Context

In a Stanford CS25 lecture, Shrimai, an AI scientist who had recently joined Mistral AI after working on NVIDIA's Nemotron models, asks whether the order a sees its data matters. Her two-phase recipe first trains on a broad, diverse mix including medium and low-quality web text, then switches to higher-quality sources such as math, code and Wikipedia, repeated more times. Quality classifiers set the mix weights, and ablation tests decide how many repeats help before gains shrink.

She reports that this model averaged 17 percent better than a baseline sampled in proportion to each source's size, and 3.4 percent better than the same quality-weighted blend shown in random order. Swapping the phases, quality first, gave weaker results. These are averages across downstream tasks from her team's NVIDIA work, as presented in the talk.

Terms in this piece · Glossary
  • Model — A computational system that uses learned patterns to produce an output from information supplied to it.
  • Training — The process that adjusts a model using examples and a measure of how well it performs.
  • pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
Read the source www.youtube.com
More from Stanford Online
Recommended reads
Comments

Checking sign-in…

Loading comments…