Intel / article

Cache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM serving

Source: together.ai
Author: Together AI
Date: Published Mar 4, 2026

Context

Together AI separates requests that need extensive prompt processing from those that can reuse earlier computation. Its cache-aware serving design gives new prompts dedicated processing capacity, while requests with reusable prefixes retrieve stored state. This prevents a follow-up conversation from waiting behind a large, unfamiliar document merely because both need prompt processing. The engineering report explains the routing and shared cache.

In synthetic coding- traffic on NVIDIA B200 GPUs, Together reports up to 35–40% more sustainable requests per second than its conventional disaggregated configurations. The gain came principally from reducing contention during prompt processing; generation throughput remained broadly comparable. The result concerns capacity under mixed traffic, rather than a uniform reduction in every response's duration.

Terms in this piece · Glossary

attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Key quotes

“By isolating heavy prefills and leveraging distributed KV cache, CPD delivers up to 40% higher sustainable throughput and significantly lower time-to-first-token (TTFT) for long-context inference — especially under mixed, real-world traffic.”
Together AI

“The large cold prompts occupy those resources for seconds at a time, and warm requests that could have been served quickly through cache reuse end up waiting in the same queue.”
Together AI

“The fastest layer lives in GPU memory, followed by host DRAM, and a cluster-wide distributed cache connected via RDMA.”
Together AI

“The 2P1D baseline reaches saturation earlier, with achieved QPS flattening around 0.75–0.8 QPS per GPU , after which queueing delays grow rapidly. In contrast, CPD continues to scale to approximately 1.1–1.15 QPS per GPU under the same workload, representing a ~40% increase in sustainable throughput before entering saturation.”
Together AI

“This highlights that as context windows grow, system-level scheduling and reuse-aware design become first-order factors in inference performance, alongside model and hardware efficiency.”
Together AI

Read the source www.together.ai