TPU🚨 is working with the popular OSS inference optimization library Mooncake on integrating TPU with Mooncake Store. Similar to NVL72, KVCache DRAM P2P pooling will initially happen on the scale-out network via TENT instead of using ICI/NVLink. 🔥 Mooncake basically improves performance per TCO of production inference!
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
Why it matters
KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition → pooling across accelerators is one of the larger levers on inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → cost, and its arrival on TPU changes the comparison between serving platforms.
postAMD code lands upstream in NVIDIA's inference transport library
postCongrats to @Zai_org on GLM-5.3. It massively beats every American open model: N