Vibeleaderboard
← All Intel
Intel / video

Stanford CS336 Language Modeling from Scratch | Spring 2025 | Overview and Tokenization

Source
youtube.com
Author
Stanford Online
Date

Context

The opening lecture of Stanford's CS336 says the course exists because researchers are drifting from the underlying technology: many now only prompt proprietary models, and the instructors say you must build a system to understand it. Frontier models are out of reach, with details undisclosed and costs enormous, so students build small ones.

They name the catch. Small models may not represent large ones. As an example, they say the share of compute spent in layers versus MLP layers (the two main building blocks of a model) shifts as models grow, so optimizing attention at small scale could target the wrong thing. They also cite emergent abilities that appear only at larger scale. So they divide knowledge into mechanics and mindset, which transfer, and intuition about data and design choices, which they say only partly transfers.

Terms in this piece · Glossary
  • attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
  • transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
Read the source www.youtube.com
More from Stanford Online
Recommended reads
Comments

Checking sign-in…

Loading comments…