Intel / video

Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer

Source: youtube.com
Author: AI Engineer
Date: Published Sep 19, 2026

From the Daily Brief

Asaf Gardin and Yuval Belfer chased a vLLM bug that corrupted roughly one in a thousand prompts with no crash or warning, only under load, and only with AI21's Jamba, which mixes attention and Mamba layers. They reproduced it on demand by dropping vLLM's GPU memory utilization from 90 percent to 20 percent at temperature zero, which made one specific request fail every time. A logprob comparison against a plain reference implementation ruled out the prefill kernel as a false lead, and threading a request ID into the forward context let them set a breakpoint and catch the scheduler running decode before prefill. Attention writes its cache before reading it, so it survives out of order scheduling, but Mamba reads its state first, so a fresh request ended up computing over a previous request's leftover state. A second, unrelated bug produced logprob spikes every twelve training steps; scaling rollouts per prompt from 8 to 128 moved the spike to step one, and the fix was changing a single index's type to size_t.

Read the 2026-09-20 Brief →

Context

Nonsensical model output can originate in the serving software's handling of a request. A debugging account for vLLM, software that runs language models, describes a scheduler starting Jamba's answer-generation phase before processing the new request's prompt.

According to the published account, Jamba's Mamba layers read stored state before replacing it. Starting generation too early therefore made a fresh request use state left from earlier work. Investigators traced the ordering error by carrying the request's identity through the computation and comparing output probabilities with a reference implementation. The case explains how a service can return apparently confident output without crashing even though its internal request state is wrong.

Key takeaways · AI-distilled

The corruption hit roughly 1 in 1,000 prompts with no crash or warning -- high-confidence gibberish -- and only appeared under load, and only with Jamba's hybrid /Mamba architecture.
The reproduction trick was dropping vLLM's GPU memory utilization from 90% to 20% at temperature zero, which made one specific request (#8,854) fail deterministically every time.
A logprob comparison against a plain reference implementation ruled out the prefill kernel as a false lead; the real find came from threading a request ID into the forward to set a breakpoint.
Why only Mamba broke: attention writes its before reading it, so out-of-order scheduling doesn't corrupt it, but Mamba reads its state first, so a fresh request ended up computing over a prior request's leftover state.
The second bug reproduced by scaling rollouts per prompt from 8 to 128, which moved the logprob spike to step one; the eventual fix was changing a single index's type to size_t.

Terms in this piece · Glossary

attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.

Read the source www.youtube.com