The Transformer Family Version 2.0
- Source
- lilianweng.github.io
- Author
- Lilian Weng
- Date
Context
Long inputs make ordinary expensive because it compares positions throughout the sequence. Doubling sequence length roughly quadruples the number of pairwise comparisons. Lilian Weng's January 2023 architecture survey explains how restricting those connections can reduce the cost.
One approach lets each position attend only to a fixed neighborhood. With that neighborhood held constant, attention cost grows linearly with sequence length. The tradeoff is that distant positions lose their direct connection in that layer. Efficient attention therefore involves deciding which information connections to retain, alongside reducing computation.
- transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
“Version 2.0 is a superset of the old version, about twice the length.”
“Later simplified Transformer was shown to achieve great performance in language modeling tasks, like in encoder-only BERT or decoder-only GPT .”
Checking sign-in…
Loading comments…