Alisa Liu kept their public LLM study notebook up to date while preparing for technical research interviews. Their account describes the method: start with Stanford's CS336 lectures to see the scope of the field, then study individual topics through papers and blog posts and code the concepts from scratch. They single out implementing and debugging a transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition →, which they say comes up often in interviews, and recommend practicing with AI assistance completely off to reproduce interview conditions. The notebook is therefore the written side of a learning process that also depended on implementation practice, not a substitute for it.
Key takeaways · AI-distilled
The notebook derives the softmax plus cross-entropy gradient as dL/dz = p - one_hot(t): the predicted distribution minus the one-hot target.
It works through activation checkpointing's memory/compute tradeoff: using K = sqrt(N) segments brings activation memory to O(sqrt N).
Its backward pass for a linear layer Z = XW + b accounts for PyTorch storing weights transposed, and explains that per-example Jacobians contract over shared tensors and stack over unshared ones.
It also includes a from-scratch AdamW with bias correction and decoupled weight decay, and the online-softmax recurrence that FlashAttention builds on.
Terms in this piece · Glossary
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
