• attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.

The insight that attention was memory-bound rather than compute-bound, and that tiling it to avoid round-trips to slow GPU memory makes it several times faster while staying exact. It is in every serving stack you touch, and the clearest worked example of why hardware-aware kernels matter.

• articleFlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware ScalingTogether AI
• articleAttention Is All You NeedAshish Vaswani et al.
• articleFast Transformer Decoding: One Write-Head is All You NeedNoam Shazeer

Sign in to comment.

Loading comments…