Frames attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → and MLPs as key-value memories with measurable interference, giving engineers a concrete mental model for why softmax sharpness and length scaling matter at long context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →.
Terms in this piece · Glossary
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
