Ultra-High Interactivity on NVIDIA GPUs? - TileRT InferenceX
- Source
- newsletter.semianalysis.com
- Author
- Bryan Shan
- Date
Context
TileRT makes a particular trade-off: faster text generation for an individual user at the expense of serving more work concurrently. Its persistent GPU pipeline reduces repeated setup and coordination during generation while overlapping computation and data movement. In SemiAnalysis measurements, an eight-NVIDIA-B200 configuration reached 340 per second per user with 8,000 input tokens and 1,000 output tokens. Benchmark report
The tested version allowed only one active request per decode node. In the report's comparison at the same input and output lengths, a GB300 configuration running at higher concurrency delivered about 240 total tokens per second per GPU, against 160.4 for TileRT, even though each user received text more slowly. The result therefore supports a specialized fast-response serving tier, with capacity planned around that concurrency constraint. Configuration and throughput comparison
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
“At batch size 1, GLM-5 at NVFP4 requires only approximately 21 GB of active-parameter traffic per generated token. The B200 HBM bandwidth roofline would therefore suggest up to 3,047 tokens/s/user without speculative decoding. In practice, GPUs come nowhere close to this limit. The gap comes from latency rather than bandwidth.”
Bryan Shan
“A CUDA graph optimizes the launching of kernels, while TileRT abolishes the kernel as the unit of execution.”
Bryan Shan
“Good enough on hardware you already own tends to beat architecturally pure on hardware you have to buy.”
Bryan Shan
“An ASIC fleet is the opposite: the ratio of speed-tier capacity to everything else is fixed in hardware the day the purchase order is signed. Changing the ratio of the physical fleet will take months to physically re-rack and re-cable.”
Bryan Shan
“That is why Cerebras serves a dense 70B at speeds no eight-GPU node can reach regardless of scheduling: software can approach the HBM roofline, but it cannot raise it.”
Bryan Shan
Checking sign-in…
Loading comments…