Cache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM serving
Source
Together AI
Author
Together AI
Date
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
prompt caching — Reusing the model's processed form of a repeated prompt prefix so subsequent calls skip re-reading it, cutting cost and latency substantially.
Why it matters
If you're serving long-context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → LLMs and fighting slow time-to-first-token, CPD shows how separating cache-warm and cache-cold workloads across prefill and decode stages can lift throughput ~40% — a concrete architectural lever beyond generic prompt cachingReusing the model's processed form of a repeated prompt prefix so subsequent calls skip re-reading it, cutting cost and latency substantially.Full definition →.
Key quotes
“By isolating heavy prefills and leveraging distributed KV cache, CPD delivers up to 40% higher sustainable throughput and significantly lower time-to-first-token (TTFT) for long-context inference — especially under mixed, real-world traffic.”
Together AI
“The large cold prompts occupy those resources for seconds at a time, and warm requests that could have been served quickly through cache reuse end up waiting in the same queue.”
Together AI
“The fastest layer lives in GPU memory, followed by host DRAM, and a cluster-wide distributed cache connected via RDMA.”
Together AI
“The 2P1D baseline reaches saturation earlier, with achieved QPS flattening around 0.75–0.8 QPS per GPU , after which queueing delays grow rapidly. In contrast, CPD continues to scale to approximately 1.1–1.15 QPS per GPU under the same workload, representing a ~40% increase in sustainable throughput before entering saturation.”
Together AI
“This highlights that as context windows grow, system-level scheduling and reuse-aware design become first-order factors in inference performance, alongside model and hardware efficiency.”
