transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
Why it matters
FlashAttention is the backbone of efficient transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition →inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → and training; the v4 co-design of pipelining, 2-CTA MMA, and a hybrid softmax path targets memory-bandwidth-bound attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → on modern GPUs, which directly affects throughput and cost for anyone serving or training large models.
Key quotes
“From the Hopper H100 to the Blackwell B200 , for instance, BF16 tensor core throughput increases from 1 to 2.25 PFLOPs, while both the SFU count and shared memory bandwidth remains unchanged.”
Together AI
“However, doing a “feeds and speeds” analysis for B200 in fact shows the opposite: the main performance bottleneck lies not in how fast the tensor cores can do MMA, but rather (a) in the SFU units for softmax exponential during the FWD computation, and (b) in the shared-memory traffic during the BWD computation.”
Together AI
“On B200 with BF16, it reaches up to 1605 TFLOPs/s (71% utilization), up to 1.3× faster than cuDNN version 9.13 and 2.7× faster than Triton.”
Together AI
“The critical detail is that exp is the bottlenecked section. We explicitly synchronize the two softmax warpgroups so they do not evaluate exp at the same time, thereby reducing MUFU contention.”
Together AI
“In practice, deterministic backward reaches up to about 85-90% of the nondeterministic throughput in our benchmarks.”
