FlashAttention is the backbone of efficient transformer inference and training; the v4 co-design of pipelining, 2-CTA MMA, and a hybrid softmax path targets memory-bandwidth-bound attention on modern GPUs, which directly affects throughput and cost for anyone serving or training large models.
As GPU throughput outpaces memory bandwidth, kernels must evolve.
We introduce FlashAttention-4, featuring new pipelining for maximum overlap, 2-CTA MMA modes to reduce shared memory traffic, and a hardware-software hybrid approach to softmax exponentials.
Transcript
As GPU throughput outpaces memory bandwidth, kernels must evolve. We introduce FlashAttention-4, featuring new pipelining for maximum overlap, 2-CTA MMA modes to reduce shared memory traffic, and a hardware-software hybrid approach to softmax exponentials.
