Intel / article

Serving DeepSeek-V4: why million-token context is an inference systems problem

Source: together.ai
Author: Together AI
Date: Published May 11, 2026

Context

DeepSeek-V4 compresses stored , but the serving engine must manage several kinds of memory to realize that saving. It combines compressed representations with exact recent context, each having different storage and reuse requirements. Together AI's early deployment account shows how retaining local state can still dominate memory consumption.

By selectively retaining local states likely to be reused, Together reports increasing total context-cache capacity on one NVIDIA HGX B200 node from roughly 1.2 million to 3.7 million . This was a storage-policy result during initial implementation. Broader throughput measurements were still forthcoming, so the concrete finding is increased cache capacity rather than a universal latency or cost advantage.

Terms in this piece · Glossary

context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.

Key quotes

“Benchmark tables miss the main point of DeepSeek-V4: the important change is architectural. V4 turns million-token context into a serving-systems problem.”
Together AI

“A full-SWA implementation actually had a higher per-token KV footprint than our V3 path — roughly 3.8 KB per token versus 3.4 KB — because the engine was storing the full sliding-window state.”
Together AI

“By keeping only the SWA states most likely to be reused, we increased total KV-cache capacity on a single NVIDIA HGX B200 node from roughly 1.2M tokens to 3.7M tokens with minimal changes.”
Together AI

“Heavily Compressed Attention (HCA) uses the same compression idea but with stride 128. At a 1M-token context length, that reduces the cache from 1M token positions to roughly 8K compressed entries.”
Together AI

“For these workloads, the cost model often shifts from price per token toward cost per completed trajectory.”
Together AI

Read the source www.together.ai