Intel / video

Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave

Source: youtube.com
Author: AI Engineer
Date: Published Sep 19, 2026

Context

CoreWeave's talk describes routing requests toward servers that already retain useful computation from earlier inputs. The publisher's account says its router considers this cached state before choosing the least-loaded server.

That matters when successive requests reuse substantial input: keeping the earlier computation available can avoid processing it again. The account also describes moving cached state to high-bandwidth storage between chat turns instead of discarding it. Together, routing and storage preserve work that a later request can reuse.

Key takeaways · AI-distilled

Roughly 80-90% of the input tokens in a typical agentic request are identical to the previous turn, which is why CoreWeave prices cached input tokens far below fresh ones — prefill, not decode, is the most expensive step to redo.
CoreWeave serves two models without forking its stack: serverless (pay-per-token, no hardware visibility, with a provisioned-throughput tier for predictable traffic) and dedicated (billed per GPU-hour behind a private gateway, where the customer picks the engine).
Batch workloads get a 12-hour window rather than real-time service, which lets CoreWeave schedule the same dedicated capacity that serves live chat traffic by day to drain batch queues overnight instead of sitting idle.
Between chat turns, CoreWeave offloads a session's to high-bandwidth storage rather than evicting it, so the next message in the conversation doesn't have to pay the prefill cost again.
The two biggest recent efficiency levers were 4-bit and speculative decoders trained asynchronously on a specific customer's own traffic data to raise acceptance length.

Terms in this piece · Glossary

inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.

Read the source www.youtube.com