Intel / article

[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback

Source: latent.space
Date: Published Oct 21, 2026

Context

DeepSeek has open-sourced DeepInfer, the inference framework behind its large-scale DeepSeek-V4.1-Flash service, under the Apache 2.0 license (earlier coverage). The repository's README credits vLLM as the project it was originally built primarily on, and it is specific about what the release covers: offline inference with the engine's core features, including draft-token speculation with confidence-scheduled verification, a bounded replay that cuts prefill work, batch-invariant and deterministic modes, and , on NVIDIA SM100 GPUs and Huawei Ascend 950 chips. It also lists what a complete service still needs and the release does not include: a system, separation of the prefill and decode stages, and load balancing across replicas. A companion repository, deepseek-recipe, handles token encoding and decoding to match the official DeepSeek API's behavior.

That missing layer matters for reading the headline numbers. They come from a Chinese-language engineering post DeepSeek published on Zhihu, which this explanation relies on through quotations and summaries rather than the original. As quoted by swyx, DeepSeek says one cluster serves 155 trillion tokens a day and earns $38 per accelerator per hour at off-peak prices. A summary by ZhihuFrontier adds that 96% of input tokens are cache hits, and that reusable prefixes are kept in a separate storage cluster of roughly 100 PiB of SSD for three days for traffic and seven days for chatbot traffic. The 155 trillion figure therefore describes a heavily cached service, counting reused input prefixes served from storage alongside newly computed work, not 155 trillion newly generated tokens, and the cache system that makes that possible is one of the components left out of the release.

The $22 million a day, roughly 80% margin and 81-day payback in this AINews issue's title are Latent Space's own arithmetic, not DeepSeek disclosures. The daily figure multiplies 155 trillion tokens by an assumed blended price, and the margin compares DeepSeek's $38 revenue figure with an assumed all-in cost of $5 to $8 per accelerator-hour; the newsletter's own analysis notes that such calculations leave out or only partly count the SSD cache fleet, CPUs, networking, idle capacity and staff. The most concrete outside check so far is also self-reported: the vLLM team says its engine, run on GB200 under DeepSeek's decode configuration, came within 6% of DeepInfer's published decode throughput, which compares one stage of serving rather than the whole service.

Key takeaways · AI-distilled

The vLLM team reported decode throughput within 6% of DeepInfer's published results on GB200 under a matched configuration; AINews stresses this is decode-stage parity, not a reproduction of DeepSeek's cache operations, routing or end-to-end latency.
Although DeepSeek calls DeepInfer fully open source, its KV-cache management and load-balancing components are not yet fully released because they remain coupled to internal infrastructure, according to the technical account summarized by AINews.
DeepInfer's SWA Bounded Replay rebuilds missing short-lived sliding-window state from only the latest 128 tokens while reusing the persistent cache. DeepSeek reports this nearly halves total prefill compute, though it is an approximation whose validity is model-specific.
DeepSeek's DSpark predicts draft-token acceptance and adjusts verification length, which DeepSeek reports improves production throughput by 6% over always verifying five draft tokens; AINews notes that is not a gain over non-.

Terms in this piece · Glossary

structured output — Forcing a model's response to match a schema, so downstream code can parse it instead of guessing at prose.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.

Read the source www.latent.space