Serving a 1.6T MoE at 271 tok/s on H20 without native FP4

DeepSeek-V4-Pro is a 1.6-trillion-parameter mixture-of-experts model released with FP8 and FP4 weights. The technical post shared by Ant Open Source, which says it built the work with the SGLang team, asks how far NVIDIA H20 GPUs can be pushed for it. H20s are widely deployed but lack native FP4 Tensor Cores; for the precisions actually used, the authors put B300 at about 45.6x the H20's peak Tensor Core compute and 1.67x its memory bandwidth. Their best observed generation rate was 271 tokens per second against 383.7 reported on B300, a ratio of 1.42x.

That 271 figure is a narrow reference point. It comes from one node of eight H20-141GB GPUs, configured to run a single request at a time with DSpark speculative decoding, in which a draft proposes tokens for the main model to verify. It is the highest result from a sweep of input lengths, reached at 256K tokens; the same setup measured 183 tok/s at 1M. Throughput was read from server logs after discarding the highest and lowest 20% of samples, and the B300 number follows the setup reported in a separate linked source rather than a side-by-side run.

The authors do not recommend serving that way. On one node, model weights and request state share the same memory, so at a 1M-token context the single-node setup has room for only one request. Their low-latency serving profile splits the model across two nodes. It reaches 150 to 174 tok/s at batch size 1, but supports batch sizes 4, 8 and 16 at 1M, 512K and 256K contexts, because spreading the weights over two nodes leaves more memory for request state. Two memory techniques enlarge that room further. Humming MXFP4AFP8 stores expert weights in the 4-bit MXFP4 format and computes with FP8 activations, shrinking weights on hardware without FP4 units. Online C128 keeps a compact aggregate KV-cache state instead of per-index state. Combined, the post reports 10.14x the per-rank token capacity of the FP8 baseline for the two-node profile and 3.88x for the high-throughput profile.

The post presents its results as a methodology rather than one benchmark, with different profiles for different jobs: optimized prefill processes a 1M-token prompt in 43.7 seconds, and a throughput-oriented profile reaches 4.67k output tokens per second per node at an average 27.4 ms per token. On accuracy, the Humming profile scored 95.5% exact-match on the GSM8K1000 math evaluation, passing the authors' 95.0% threshold; the public no-degradation comparison they cite was run on DeepSeek-V4-Flash, so they treat it only as an external reference for V4-Pro. The announcement says the Humming and DSpark optimizations are already upstreamed, while the post says the pipeline-parallel DSpark integration is still being upstreamed.