• Roughly 80-90% of the input tokens in a typical agentic request are identical to the previous turn, which is why CoreWeave prices cached input tokens far below fresh ones — prefill, not decode, is the most expensive step to redo.
• CoreWeave serves two models without forking its stack: serverless (pay-per-token, no hardware visibility, with a provisioned-throughput tier for predictable traffic) and dedicated (billed per GPU-hour behind a private gateway, where the customer picks the engine).
• Batch workloads get a 12-hour window rather than real-time service, which lets CoreWeave schedule the same dedicated capacity that serves live chat traffic by day to drain batch queues overnight instead of sitting idle.
• Between chat turns, CoreWeave offloads a session's KV cache to high-bandwidth storage rather than evicting it, so the next message in the conversation doesn't have to pay the prefill cost again.
• The two biggest recent efficiency levers were 4-bit quantization and speculative decoders trained asynchronously on a specific customer's own traffic data to raise acceptance length.

• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
• streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
• KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
• quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.

Details how KV-cache locality and workload shape (agentic vs. batch vs. streaming) drive real infrastructure and pricing tradeoffs, useful for teams choosing or building inference platforms.

• videoOperating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta
• videoAre LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google
• videoRouting LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI
• videoWhat's New in Inference Engineering — Philip Kiely, Baseten

• videoKV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red HatAI Engineer
• videoOperating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, MetaAI Engineer
• videoWhat's New in Inference Engineering — Philip Kiely, BasetenAI Engineer

Checking sign-in…

Loading comments…