Intel / article

Ultra-High Interactivity on NVIDIA GPUs? - TileRT InferenceX

Source: Bryan Shan
Author: Bryan Shan
Date: Published Aug 10, 2026

Terms in this piece · Glossary

inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Why it matters

Explains what actually limits interactive latency, so you can reason about when a fast mode is worth paying for and where alternative hardware wins.

Key quotes

“At batch size 1, GLM-5 at NVFP4 requires only approximately 21 GB of active-parameter traffic per generated token. The B200 HBM bandwidth roofline would therefore suggest up to 3,047 tokens/s/user without speculative decoding. In practice, GPUs come nowhere close to this limit. The gap comes from latency rather than bandwidth.”
Bryan Shan

“A CUDA graph optimizes the launching of kernels, while TileRT abolishes the kernel as the unit of execution.”
Bryan Shan

“Good enough on hardware you already own tends to beat architecturally pure on hardware you have to buy.”
Bryan Shan

“An ASIC fleet is the opposite: the ratio of speed-tier capacity to everything else is fixed in hardware the day the purchase order is signed. Changing the ratio of the physical fleet will take months to physically re-rack and re-cable.”
Bryan Shan

“That is why Cerebras serves a dense 70B at speeds no eight-GPU node can reach regardless of scheduling: software can approach the HBM roofline, but it cannot raise it.”
Bryan Shan

Read the source newsletter.semianalysis.com