Stanford CS336 Language Modeling from Scratch | Spring 2025 | Overview and Tokenization
- Source
- youtube.com
- Author
- Stanford Online
- Date
Context
The opening lecture of Stanford's CS336 says the course exists because researchers are drifting from the underlying technology: many now only prompt proprietary models, and the instructors say you must build a system to understand it. Frontier models are out of reach, with details undisclosed and costs enormous, so students build small ones.
They name the catch. Small models may not represent large ones. As an example, they say the share of compute spent in layers versus MLP layers (the two main building blocks of a model) shifts as models grow, so optimizing attention at small scale could target the wrong thing. They also cite emergent abilities that appear only at larger scale. So they divide knowledge into mechanics and mindset, which transfer, and intuition about data and design choices, which they say only partly transfers.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
videoStanford Webinar - Building Trustworthy AI: Navigating Security in the Agentic Era
videoStanford CS547 HCI Seminar | Autumn 2025 | HCI Systems in the Age of AI Code Generation
videoStanford CS25: Transformers United V6 I Advancing Science and Medicine with Collaborative AI Agents
videoStanford CS153 Frontier Systems | Andreas Blattmann from Black Forest Labs on Visual Intelligence
videoStanford CS25: Transformers United V6 I From Language Models to Native Multimodal IntelligenceStanford Online
videoStanford CS25: V5 I On the Biology of a Large Language Model, Josh Batson of AnthropicStanford Online
videoStanford CS25: Transformers United V6 I From Next-Token Prediction to Next-Generation IntelligenceStanford Online
Checking sign-in…
Loading comments…