Intel / article

DeepSeek open-sources DeepInfer, its production inference stack

Source: github.com
Author: deepseek_ai
Date: Published Oct 20, 2026

Context

DeepSeek has open-sourced DeepInfer, which its README describes as the inference framework that powers the company's large-scale production services. What has been published is the engine that runs the model, not the whole serving system. The release provides offline inference, meaning a script loads the model and generates outputs directly rather than answering requests as a hosted service. The README lists three things a complete inference service would still need that this release leaves out: a system, a service-level store that keeps the state of already-processed tokens available for reuse so it does not have to be recomputed; disaggregation, which splits the VE, prefill and decode stages onto separate workers; and load balancing across data-parallel replicas. For disaggregation, it adds that most of the required engine functionality is already implemented.

The scope is narrow. DeepInfer supports one model, DeepSeek-V4.1-Flash, and two kinds of hardware: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. When DeepSeek launched V4.1-Flash, it described the model as a 552-billion-parameter design and said it would work with the open-source community on inference support. DeepInfer is DeepSeek's own engine for that model. Its feature list reflects that design. Expert parallelism with load balancing (EPLB) spreads the model's experts across devices and evens out their load. A scheme called DSpark proposes draft tokens and then uses confidence estimates to schedule when they are verified. Batch-invariant inference keeps a request's results identical no matter which other requests share its batch, and deterministic sampling returns the same output for the same input tokens and sampling settings. The README marks both of those as NVIDIA-only.

DeepSeek's announcement says DeepInfer runs V4.1-Flash at scale with high throughput and low latency, and frames the release as a way to help others lower compute costs and energy use. The README acknowledges that the engine was originally built primarily on vLLM, the open-source serving engine that, according to SemiAnalysis, ran V4.1-Flash on day one on NVIDIA hardware. DeepInfer's own speed results appear only as prefill and decode charts, without figures in the README text. They were measured on 4 SM100 GPUs for prefill and 64 for decode, and the README points to a script for reproducing them.

Key takeaways · AI-distilled

DeepInfer speeds decoding with DSpark draft tokens and uses confidence estimates to schedule their verification; the default maximum draft length is 5, set through DEEPINFER_USE_NEXTN_DECODE.
Deterministic sampling requires batch-invariant mode on, variable-length MTP off and a fixed seq_seed in SamplingParams, and both batch invariance and deterministic sampling are NVIDIA-only in this release.
The release supports only DeepSeek-V4.1-Flash, on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. A complete service still needs a KV cache system, prefill/decode disaggregation and load balancing across data-parallel replicas, none of which are included.
SWA bounded replay reduces decoder prefill work by replaying only a bounded suffix of tokens. DeepSeek's default V4.1 setting keeps 140 tokens, and the retained suffix must cover the sliding window plus the maximum backtracking length.
DeepSeek says DeepInfer was originally built mainly on vLLM, with ideas from SGLang and TensorRT-LLM, and its kernels include FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels.

Terms in this piece · Glossary

KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.

Read the source github.com