A kernel-centric path to real-time video generation on Trainium
Source
www.amazon.science
Date
Key takeaways · AI-distilled
Amazon's Neuron Science team and Reactor ported Rolling Forcing, an autoregressive diffusion video model, to Trainium; the hard constraint is streamingSending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.Full definition → frames at 16 fps or better, since a late frame breaks playback.
With hand-written NKI kernels, the team reports 3D rotary embeddingA list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.Full definition → dropped from five seconds to 1.8 ms, KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition → copies from 23 ms to 1.9 ms per layer, and attention transposes were fused away into the attention kernel.
The model has 12 attention heads but a chip has 8 Neuron cores, so tensor parallelism alone needed wasteful padding and sequence parallelism alone could split frames; the team split heads across 4 cores and sequences across 2.
The cache-update pass has far less compute than denoising, so running it as its own round underused the chips; the team batched the components both phases share and split only where they differ.
The optimized pipeline used 11 GB of high-bandwidth memory where the standard eager-mode path ran out of memory, and the NKI-Dev-Suite AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → produced a working 3D-RoPE kernel on its first attempt.
Terms in this piece · Glossary
streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
Amazon Science details the systems engineering (kernel-level optimization) needed to serve real-time interactive video world models on Trainium, a concrete constraint on how such models can be deployed at low latency and scale.
