AMD's MI355X on day one could only run a non-native FP8 checkpoint and served 1-2 tokens per user per second — slower than people read. Technically working, practically unusable; kernel work made it 100x faster by day 26.
AMD's ATOM engine had the KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition → hardcoded to a single sequence slot, so a second concurrent request had nowhere to store its state. Effective batch size: one user, no matter the hardware.
Nvidia's own TensorRT-LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → stumbled on the model and SemiAnalysis had to patch its kernel launch code. The open engines, vLLM and SGLang, ran the native checkpoint on release day without drama.
speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition → via multi-token prediction landed on day 3 from SGLang and lifted throughput most at high interactivity, where small-batch decode is memory-bound rather than compute-bound.
The CUDA moat shows up in distributed inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition →, not single-node: multi-node disaggregated prefill worked on GB200 at launch, and that recipe beat a B200 run by up to 5x at low-interactivity settings.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
Why it matters
If you're deciding which inference engine or GPU to deploy DeepSeek v4 on, this tracks real Day-0-to-week-4 performance evolution with specific bug reports and kernel fixes across vLLM, SGLang, TensorRT-LLM, and hardware from NVIDIA, AMD, and Huawei — letting you time your deployment around when a stack actually becomes usable rather than trusting vendor marketing.
Key quotes
“Huawei has proven it can sling a stone on Day 0; whether it can fell a moving giant is yet to be seen.”
“ATOM, by the way, currently has zero production customers.”
