KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat
Source
AI Engineer
Author
AI Engineer
Date
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
Agentic traffic runs 90%+ cache hit rates and 100:1 input-output ratios, so routing to the pod holding the cache beats buying GPUs; splitting prefill from decode cuts P99 inter-tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → latency from ~900ms to ~100ms, but only with an RDMA fabric.
