attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
Why it matters
A clear, intuition-first walkthrough of query-key-value attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition →, multi-head, and multi-layer mechanics — useful if you want to actually understand why the transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition → works rather than just memorize the diagram.
Key quotes
“Via this approach, the encoder had to cram the entire input into a fixed-size vector which is then passed to the decoder—this single vector had to convey everything about the input sentence!”
Eugene Yan
“Because we use the softmax function in attention, it amplifies the highest value while squashing the lower ones. As a result, each head tends to focus on a single element.”
Eugene Yan
“Because attention acts as a filter, it blocks most information from passing through.”
Eugene Yan
“The Transformer is a magnificient neural network architecture because it is a general-purpose differentiable computer.”
Andrej Karpathy
“Stacking attention layers also broadens the model’s receptive field. The first attention layer produces context vectors by attending to interactions between pairs of words in the input sentence. Then, the second layer produces context vectors based on pairs of pairs, and so on.”
