Smaller, faster, safer: running Kimi and GLM at scale

Cloudflare reports higher total output from a Kimi K2.6 deployment by shrinking its conversation cache enough to serve more requests simultaneously. The cache stores intermediate attention data used during text generation. In its experiment, changing that data from 16-bit to 8-bit representation halved its memory footprint.

At the same tested request counts, the compressed cache was slower. Its advantage appeared when memory became the limit: it supported 64 concurrent requests and reached 2,192 output tokens per second, versus 1,558 at the uncompressed configuration's peak of 32 requests. The reported 41% gain is therefore aggregate serving throughput from higher concurrency, rather than a 41% speedup for each individual conversation.

“On Kimi K2.6, that raises the amount of context we can hold in memory from roughly 686,000 tokens to about 1.37 million, twice as much.”

Cloudflare

“BF16 runs out of cache at 32 concurrent requests and can't admit a 33rd, while FP8 keeps going to 64 and reaches 2,192 tokens per second, about 41% higher than BF16's peak, for roughly 30% less cost per token.”

Cloudflare

“The checkpoint shrinks from 705 GB to 421 GB, about 40%, and per-GPU memory across an 8-way tensor-parallel deployment drops from roughly 88 GB to 52 GB, which leaves room for around 1.18 million tokens of KV cache on the same hardware.”

Cloudflare

“at our request volumes, even a one-in-a-billion mistake would show up regularly.”

Cloudflare

“The cost is under 1% on both throughput and tail latency, and even the upper bound of the 95% confidence interval stays near 1%.”

Cloudflare