KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat

Agentic traffic runs 90%+ cache hit rates and 100:1 input-output ratios, so routing to the pod holding the cache beats buying GPUs; splitting prefill from decode cuts P99 inter-token latency from ~900ms to ~100ms, but only with an RDMA fabric.

• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
• AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
• token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

• videoAI Security Engineer Foundations + Certificate — Micah Silverman, Snyk
• videoThe SDLC You Build Yourself — Andy Wong, Andrei Bocan & Shane Wolf, Atlassian
• videoAgent Speedrun — Elizabeth Fuentes Leone & Sandhya Subramani, AWS
• videoCan Your Agent Hear You Now? Building Live Voice Agents with Gemini — Thor Schaeff

• videoVertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeaveAI Engineer
• videoWhat's New in Inference Engineering — Philip Kiely, BasetenAI Engineer
• videoAgents That Own Their Inference — Du'an Lightfoot & Khaja Omer, Akamai TechnologiesAI Engineer

Checking sign-in…

Loading comments…