Mooncake Store brings KV cache pooling to TPU over the scale-out network
- Source
- x.com
- Date
TPU🚨 is working with the popular OSS inference optimization library Mooncake on integrating TPU with Mooncake Store. Similar to NVL72, KVCache DRAM P2P pooling will initially happen on the scale-out network via TENT instead of using ICI/NVLink. 🔥 Mooncake basically improves performance per TCO of production inference!

Context
SemiAnalysis reported on August 19 that work is under way to connect Google's TPU chips to Mooncake Store, and that pooling cached model state in server memory would at first run over the ordinary data-center network through Mooncake's TENT transfer engine rather than over the TPUs' own chip-to-chip links. Mooncake's public repository confirms TPU support work in TENT and gives the reasoning for that network choice. In the material read, the TPU code so far sits in the transfer engine rather than in Mooncake Store, and TPU transfers cannot yet run on real hardware.
The cache in question is the : intermediate data a computes for text it has already processed, which a server can keep and reuse instead of recomputing when the same prefix comes back. Mooncake is an open-source serving project built around that cache. Its README describes using "otherwise underutilized CPU, DRAM, and SSD resources" in GPU clusters to build a shared KV cache pool. Mooncake Store is the part that holds reusable KV caches and model weights across a cluster, and it sits on Mooncake's transfer engine, which moves data over TCP, RDMA and other transports. TENT is a transfer engine from the same project, which its technical report describes as choosing a network path for each slice of data at the moment it is sent.
A June design proposal in the Mooncake repository explains why TPU traffic takes the network route. It proposed adding TPUs to TENT so KV cache could move between TPU machines that handle the prompt (prefill) and those that generate output (decode). Unlike Nvidia GPUs, it says, TPUs do not give third-party network cards access to their high-bandwidth memory, so data must be copied through host memory first. The proposal adds that this route avoids flooding the TPUs' Inter-Chip Interconnect (ICI) with asynchronous KV cache traffic, preserving its bandwidth for the synchronous collective communication that model-parallel computation depends on. The first implementation, merged on July 4, follows that plan: a copy from TPU memory to host memory through the PJRT runtime interface, then host to host over RDMA or TCP. It is off by default and was tested with a mock rather than TPU hardware.
The rest is less settled. On September 1 the proposal's author wrote that they were "working with TPU team on long-term support for mooncake." A follow-up pull request opened on November 10 and, as of November 11, still awaiting review says that in the merged code "only a test mock" implements the device-copy interface, "so TPU transfers cannot run on real hardware yet," and does not touch Mooncake Store. None of the pages read contains an official statement from Google, a timeline or performance figures. The post's comparison with Nvidia's NVL72 systems and its claim that Mooncake improves performance per total cost of ownership are SemiAnalysis's own assessments.
- KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Checking sign-in…
Loading comments…




