token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
Why it matters
If you're serving million-tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →-context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → workloads, this breaks down the real bottlenecks — compressed KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition → layouts, prefix caching, and kernel maturity on HGX B200 — so you can pick endpoint profiles and reason about long-context inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → cost/latency instead of treating context length as a free model feature.
Key quotes
“Benchmark tables miss the main point of DeepSeek-V4: the important change is architectural. V4 turns million-token context into a serving-systems problem.”
Together AI
“A full-SWA implementation actually had a higher per-token KV footprint than our V3 path — roughly 3.8 KB per token versus 3.4 KB — because the engine was storing the full sliding-window state.”
Together AI
“By keeping only the SWA states most likely to be reused, we increased total KV-cache capacity on a single NVIDIA HGX B200 node from roughly 1.2M tokens to 3.7M tokens with minimal changes.”
Together AI
“Heavily Compressed Attention (HCA) uses the same compression idea but with stride 128. At a 1M-token context length, that reduces the cache from 1M token positions to roughly 8K compressed entries.”
Together AI
“For these workloads, the cost model often shifts from price per token toward cost per completed trajectory.”
