The notebook derives the softmax plus cross-entropy gradient as dL/dz = p - one_hot(t): the predicted distribution minus the one-hot target.
It works through activation checkpointing's memory/compute tradeoff: using K = sqrt(N) segments brings activation memory to O(sqrt N).
Its backward pass for a linear layer Z = XW + b accounts for PyTorch storing weights transposed, and explains that per-example Jacobians contract over shared tensors and stack over unshared ones.
It also includes a from-scratch AdamW with bias correction and decoupled weight decay, and the online-softmax recurrence that FlashAttention builds on.
Terms in this piece · Glossary
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
Why it matters
Alisa Liu's public notebook derives the mechanics practitioners rely on, like the softmax+cross-entropy gradient, online-softmax numerics, and activation-checkpointing memory tradeoffs, in one place as a reference for building or debugging LLM training code.
