Intel / article

A kernel-centric path to real-time video generation on Trainium

Source: amazon.science
Date: Published Sep 25, 2026

Context

Amazon scientists and Reactor, a platform for deploying real-time interactive AI models, worked to run video generation on Trainium, AWS's AI hardware. Their engineering account uses Rolling Forcing, a model that generates video frame by frame while keeping a rolling cache of recent . Every layer reads and rewrites that cache at each step, so small delays repeat on every pass and can add up to a missed frame deadline.

The team wrote hardware-specific routines (kernels) with the Neuron Kernel Interface for the costly operations. Amazon's Jun Wu is quoted reporting cache copies falling from 23 to 1.9 milliseconds per layer, and attention transposes eliminated by fusing them into the attention computation. These are the team's reported measurements for this pipeline.

Key takeaways · AI-distilled

Amazon's Neuron Science team and Reactor ported Rolling Forcing, an autoregressive diffusion video model, to Trainium; the hard constraint is streaming frames at 16 fps or better, since a late frame breaks playback.
With hand-written NKI kernels, the team reports 3D rotary dropped from five seconds to 1.8 ms, copies from 23 ms to 1.9 ms per layer, and attention transposes were fused away into the attention kernel.
The model has 12 attention heads but a chip has 8 Neuron cores, so tensor parallelism alone needed wasteful padding and sequence parallelism alone could split frames; the team split heads across 4 cores and sequences across 2.
The cache-update pass has far less compute than denoising, so running it as its own round underused the chips; the team batched the components both phases share and split only where they differ.
The optimized pipeline used 11 GB of high-bandwidth memory where the standard eager-mode path ran out of memory, and the NKI-Dev-Suite agent produced a working 3D-RoPE kernel on its first attempt.

Terms in this piece · Glossary

streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.

Read the source www.amazon.science