Chasing the batch-1 floor: Ling-3.0-flash with DSpark on Blackwell

Ant Open Source shared a joint engineering post from the RadixArk SGLang team and Ant's Ling infrastructure team about serving inclusionAI's Ling-3.0-flash to one user at a time on 4 NVIDIA Blackwell GPUs. At batch size 1 there is no larger batch to absorb fixed overheads, so every delay is felt directly. Both paths in the post use speculative decoding: a small draft model proposes a run of tokens, and the full model checks them all in one pass. The post frames time per output token (TPOT) as roughly step time divided by the average number of tokens accepted per step, which gives two levers: make each step faster, or get more tokens accepted per step.

Most of the post is about the first lever, on the existing NEXTN draft path. The largest structural fix was a host-side stall: one attention backend inherited a default that forced the CPU to wait for the previous GPU step every iteration, costing a median 485 microseconds per step. Declaring that it did not need the value let the CPU queue work ahead of the GPU. Chained kernel launches, two kernel fusions and running the router gate and output projection in bf16 instead of fp32 (worth roughly +10%) followed. Together these moved single-request throughput from 288 to 606 tok/s and mean TPOT from 3.33 to 1.53 ms, although the 288 baseline was a shorter campaign checkpoint rather than part of the 1,000-request comparison.

DSpark works on the second lever. The post says the DSpark algorithm itself is public; Ant's contribution was adapting it to Ling, distilling the draft mainly on Ling's own post-training data, choosing the draft architecture by ablation, and adding a loss term aimed at longer accepted prefixes. Serving it hit a familiar problem: the first trace showed 47% GPU idle because FlashInfer planning calls were blocking on reads from GPU memory, which the team removed by computing those inputs on the host. In the controlled comparison, with the same 1,000 requests on the same machine, accept length rose from 3.25 to 9.95 and mean TPOT fell from 1.53 to 0.78 ms, 1.9x lower, at 1,120 tok/s.

The figures are narrower than a general model speed claim. Every headline run used a synthetic random workload of 8,192 input and 1,024 output tokens, and the authors say accept length depends on the prompt and output distribution, so 9.95 describes this workload rather than the model. They also note the 9.95 result was measured with a block size 16 DSpark draft, while the released checkpoint uses block size 8, so the published checkpoint may not reproduce that number. The Hugging Face and ModelScope model pages could not be read for this explanation. Ant Ling's own release thread announced the same checkpoint with the same 1,120 tok/s and 0.78 ms figures, which belong to this Ling adaptation rather than to DSpark in general.