The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
For each token, attention scores how relevant every other token is and blends information accordingly — resolving what "it" refers to, connecting a variable to its definition 200 lines up. Stacked in layers, this simple weighted-lookup builds surprisingly deep understanding.
Its cost grows with the square of input length, since every token attends to every other. That quadratic bill is why long context is expensive and why FlashAttention, sliding windows, and sparse-attention papers keep appearing in the index.