InferenceX DeepSeek v4 Performance Tracker

Context

• AMD's MI355X on day one could only run a non-native FP8 checkpoint and served 1-2 tokens per user per second — slower than people read. Technically working, practically unusable; kernel work made it 100x faster by day 26.
• AMD's ATOM engine had the KV cache hardcoded to a single sequence slot, so a second concurrent request had nowhere to store its state. Effective batch size: one user, no matter the hardware.
• Nvidia's own TensorRT-LLM stumbled on the model and SemiAnalysis had to patch its kernel launch code. The open engines, vLLM and SGLang, ran the native checkpoint on release day without drama.
• Speculative decoding via multi-token prediction landed on day 3 from SGLang and lifted throughput most at high interactivity, where small-batch decode is memory-bound rather than compute-bound.
• The CUDA moat shows up in distributed inference, not single-node: multi-node disaggregated prefill worked on GB200 at launch, and that recipe beat a B200 run by up to 5x at low-interactivity settings.

• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
• KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
• speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.

• articleThe Huawei-DeepSeek Symbiosis
• articleEngrams Embedding Entendre: Codesign for Efficient DRAM/SSD Offloading
• articleVera Rubin NVL72 Agentic Inference: 67x better Performance per Dollar
• articleUltra-High Interactivity on NVIDIA GPUs? - TileRT InferenceX

• articleDeepSeek open-sources DeepInfer, its V4.1-Flash inference stackdeepseek_ai
• articleThe Huawei-DeepSeek SymbiosisBryan Shan
• articleDeepseek V4 Flash 0731 Scores 50 On The Artificial Analysis Intelligence Index 10 Points Above Previous Deepseek V4 FlashArtificial Analysis articles

Checking sign-in…

Loading comments…