FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Source
arxiv.org
Author
Tri Dao et al.
Date
Context
FlashAttention speeds up attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → by reducing data movement inside a GPU. Attention lets a transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition → combine information from different positions in a sequence. The method divides that calculation into tiles, reducing transfers between the GPU's large memory and its smaller on-chip memory while preserving exact attention. The 2022 paper reports faster training across several tested workloads. Its central contribution is showing that the arrangement of memory reads and writes can materially improve execution speed, even when the attention calculation itself remains exact.
Terms in this piece · Glossary
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.