Intel / article

Five Ways NVIDIA Co-Design Maximizes the Value of Every Byte in the AI Factory

Source: developer.nvidia.com
Author: Shara Tibken
Date: Published Nov 5, 2026

Context

NVIDIA's case for its Vera Rubin NVL72 rack is that memory capacity is no longer the limit on serving AI agents. The question is how much useful work each byte does. Rubin GPUs keep the same 288 GB of high-bandwidth memory (HBM, the fast memory stacked beside the chip) as the previous GB300, yet NVIDIA says the rack delivers up to 30 times GB300's throughput per megawatt on SemiAnalysis's AgentX benchmark at 160 tokens per second per user. Its co-design post attributes that to five changes across the chip, interconnect, serving software, CPU and storage.

Most of the five deal with the , the stored results for text a model has already processed, which lets it continue a conversation without reprocessing the whole history. Agents generate a lot of reusable , such as system prompts, tool descriptions and task history, and paying to recompute it wastes both memory and time. NVFP4, a 4-bit number format, shrinks model weights about 3.5 times compared with 16-bit and can halve the cache compared with 8-bit. Spreading a model's specialist sub-networks across many GPUs leaves each GPU more room for cache. Dynamo, NVIDIA's serving software, sends requests to the GPU that already holds their context and can restore cache kept outside HBM instead of recomputing it.

The other two levers add places to keep cache outside the GPU. While an agent waits on a tool call or code run, its cache sits idle in HBM and risks being evicted. NVIDIA's Vera CPU has up to 1.5 TB of memory linked to the GPU at 1.8 TB/s, so the paused session's cache can be parked there and brought back when the agent resumes. For context needed across turns, agents or machines, a new CMX storage tier of shared network-attached flash lets a prefix computed on one node be reused across the pod, which NVIDIA says gives up to 5 times the tokens per second of traditional storage for long-context inference.

The post explains a mechanism but does not measure how much of the 30x comes from each lever, and bandwidth also rose between generations. NVIDIA first reported the figure in its August AgentX post, which said the Vera Rubin results were measured by NVIDIA on a DeepSeek V4-Pro workload and were pending SemiAnalysis review. Treat it as NVIDIA's best case for one model at one speed, not a general gain.

Key takeaways · AI-distilled

NVFP4 stores each value as a 4-bit E2M1 element, with an FP8 scale factor shared by every 16-value micro-block and a per-tensor FP32 scale on top; NVIDIA says this two-level scaling preserves accuracy where naive 4-bit formats fall short.
NVIDIA says storing the KV cache itself in NVFP4 cuts its footprint by up to 50% versus FP8, keeping roughly twice as much context on the GPU, which raises prefill cache-hit rates and lowers bandwidth use during decode.
Spreading MoE experts widely only pays off with a fast fabric: on GB200 NVL72, NVIDIA reports EP32 delivers up to 1.8x more output tokens per second per GPU than EP8 at 100 tokens/s per user, because 130 TB/s of NVLink absorbs the all-to-all traffic.
NVIDIA argues a slow CPU costs HBM twice in agentic inference: idle KV cache sits in GPU memory during tool calls, then may be evicted and have to be rebuilt by a full prefill when the tool returns.
NVIDIA positions CMX as a new G3.5 tier between node-local SSDs and enterprise storage, since shared storage carries replication and metadata overhead that ephemeral KV cache does not need, and claims up to 5x tokens per second and power efficiency versus traditional storage.

Terms in this piece · Glossary

KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.

Read the source developer.nvidia.com