Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer
Source
AI Engineer
Author
AI Engineer
Date
From the Daily Brief
Asaf Gardin and Yuval Belfer chased a vLLM bug that corrupted roughly one in a thousand prompts with no crash or warning, only under load, and only with AI21's Jamba, which mixes attention and Mamba layers. They reproduced it on demand by dropping vLLM's GPU memory utilization from 90 percent to 20 percent at temperature zero, which made one specific request fail every time. A logprob comparison against a plain reference implementation ruled out the prefill kernel as a false lead, and threading a request ID into the forward context let them set a breakpoint and catch the scheduler running decode before prefill. Attention writes its cache before reading it, so it survives out of order scheduling, but Mamba reads its state first, so a fresh request ended up computing over a previous request's leftover state. A second, unrelated bug produced logprob spikes every twelve training steps; scaling rollouts per prompt from 8 to 128 moved the spike to step one, and the fix was changing a single index's type to size_t.
The corruption hit roughly 1 in 1,000 prompts with no crash or warning -- high-confidence gibberish -- and only appeared under load, and only with Jamba's hybrid attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition →/Mamba architecture.
The reproduction trick was dropping vLLM's GPU memory utilization from 90% to 20% at temperature zero, which made one specific request (#8,854) fail deterministically every time.
A logprob comparison against a plain reference implementation ruled out the prefill kernel as a false lead; the real find came from threading a request ID into the forward context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → to set a breakpoint.
Why only Mamba broke: attention writes its KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition → before reading it, so out-of-order scheduling doesn't corrupt it, but Mamba reads its state first, so a fresh request ended up computing over a prior request's leftover state.
The second bug reproduced by scaling rollouts per prompt from 8 to 128, which moved the logprob spike to step one; the eventual fix was changing a single index's type to size_t.
Terms in this piece · Glossary
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
videoWhat's New in Inference Engineering — Philip Kiely, Baseten
videoLarge clusters for small models — Daniel Svonava, Superlinked
videoWeight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli
videoThe Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI
