Attention? Attention!
- Source
- lilianweng.github.io
- Author
- Lilian Weng
- Date
Context
lets a translation model draw on different parts of the source sentence as it generates each output word. Lilian Weng's 2018 tutorial contrasts this with an early design that compressed the entire input into one fixed-size summary, creating a bottleneck for long sentences.
With attention, the encoder retains representations of source positions. For each output step, the decoder assigns weights to those representations and combines them into a specific to that step. Different output words can therefore use different mixtures of source information. The useful change is renewed access to the encoded sentence throughout generation, easing dependence on a single summary.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
“A critical and apparent disadvantage of this fixed-length context vector design is incapability of remembering long sentences. Often it has forgotten the first part once it completes processing the whole input.”
Lilian Weng
“Rather than building a single context vector out of the encoder’s last hidden state, the secret sauce invented by attention is to create shortcuts between the context vector and the entire source input.”
Lilian Weng
“With the help of the attention, the dependencies between source and target sequences are not restricted by the in-between distance anymore!”
Lilian Weng
“It presented a lot of improvements to the soft attention and make it possible to do seq2seq modeling without recurrent network units.”
Lilian Weng
Checking sign-in…
Loading comments…