Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference
Source
Tanya Lenz
Author
Tanya Lenz
Date
Terms in this piece · Glossary
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
As AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → grows, attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → becomes the dominant cost term, which changes both model choice and serving design.
Key quotes
“As agentic and long-context workloads become common, the context lengths increase and attention consumes a larger share of inference time (Figure 1). Because attention now dominates that cost, how it is designed—not just how it is implemented—increasingly determines a model’s inference performance.”
Tanya Lenz
“The ridge point marks the transition from memory-bound to compute-bound. Prefill lies well above it and is compute-bound, while decode lies below it and is memory-bound (Figure 2).”
Tanya Lenz
“A partially filled tile costs as much as a full tile, so Hsz = 64 pays for 128. Hsz ≥ 512 pushes close to the tensor memory (TMEM) capacity limit. That makes 128 and 256 the efficient choices.”
Tanya Lenz
“TP has a practical limit: KV heads must divide evenly across GPUs. Once TP > KH, a group’s query heads span multiple ranks, each requiring a copy of the shared KV head. This duplicates KV state, adding memory and bandwidth overhead without benefit (Figure 7).”
Tanya Lenz
“The use case sets sequence length, but the cost is asymmetric: prefill grows as ISL², while decode grows linearly with KVSL.”
articleHow to Use AI Agents to Prepare 3D Scenes for Simulation
articleTranslating CUDA Tile Operations from Python to Rust Using Agentic AI
articleHow NVIDIA Groq 3 LPX Deterministic Execution Drives Power-Efficient High-Interactivity Inference on NVIDIA Vera Rubin
articleAccelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine
