Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang
Source
blog.cloudflare.com
Date
Why it matters
Shared KV cache bugs in inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → engines can return other tenants' data on very long prompts. The report details scope, timeline and SGLang hardening, which shows what to ask of hosted inference providers.
Key takeaways · AI-distilled
Cloudflare served GLM 5.3 with a 1,048,576-tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → but only 681,280 tokens of prefill KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition →, so 155 responses to longer prompts between Aug 27 and Sep 9 included data from other customers' recent requests.
SGLang's disaggregated mode swapped an over-length prompt for a one-token placeholder but still handed it to decode, which then attended over reserved KV pages that still held earlier users' data and returned HTTP 200.
Cloudflare's pre-launch tests checked only status codes and speed at long contexts, not whether outputs were coherent, so the known pool-size mismatch looked safe; it now checks pool size against context length from the running engine.
Cloudflare tells operators running disaggregated SGLang to upgrade to v0.5.22 or later, which rejects over-length requests before any KV transfer, and to measure KV pool size against context length after every config change.
Cloudflare tested a page-tagging check that fails a request about to read a KV page it does not own, at under 1% throughput cost, but disabled it over reliability issues and is building a general version with the SGLang team.
Terms in this piece · Glossary
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
