transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
Why it matters
A rigorous, implementation-backed walkthrough of attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → and the transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition → architecture that grounds engineers in the mechanics behind modern LLMs, with code links for self-attention, Pointer Networks, and Neural Turing Machines.
Key quotes
“A critical and apparent disadvantage of this fixed-length context vector design is incapability of remembering long sentences. Often it has forgotten the first part once it completes processing the whole input.”
Lilian Weng
“Rather than building a single context vector out of the encoder’s last hidden state, the secret sauce invented by attention is to create shortcuts between the context vector and the entire source input.”
Lilian Weng
“With the help of the attention, the dependencies between source and target sequences are not restricted by the in-between distance anymore!”
Lilian Weng
“It presented a lot of improvements to the soft attention and make it possible to do seq2seq modeling without recurrent network units.”
