Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine

When an AI agent's call to a language model times out, its client usually gives up and sends the same request again; the OpenAI Python SDK retries twice by default. In a preprint, Xindan Zhang of the University of Toronto and Fei Wang of the University of British Columbia Okanagan show that on the vLLM serving engine these retries can keep a server stuck after a brief traffic spike has passed, and that the obvious fix, telling the engine to stop working for clients who have hung up, did not restore service in any of their runs. They tested three open models (Qwen2.5-7B-Instruct, Qwen3-14B and Llama-3.1-8B-Instruct), each on one H100 GPU, with synthetic prompts of about 12,000 tokens. Their client abandons an attempt after 120 seconds and tries up to four times, and traffic runs at 0.6 of the engine's measured capacity except for a 107-second burst at 2.7 times capacity.

Retries alone explain why the engine could not catch up. Calls arriving at 0.6 of capacity that each make four attempts add up to 2.4 times what the engine can finish. With retries off, all three models recovered within the 12 minutes after the burst. With retries on and no cancellation, none of the calls arriving in that period was delivered, because each abandoned attempt kept running with no one waiting for it. Passing the client's disconnect through so that vLLM aborted the abandoned work helped only a little: about 12 to 38 percent of those calls were delivered, and service did not recover within those 12 minutes.

Scheduler traces of the Qwen2.5-7B model suggest why cancellation fell short. Every abort reached vLLM within half a second of the client leaving, but the queue stayed about one deadline long. A retry that vLLM ran therefore typically started processing its prompt less than half a second before its own 120-second limit expired, was aborted in turn, and sent its call to the back of the queue again. Executions that were later aborted took 79 to 85 percent of the prompt tokens the engine computed, and part of that work was already issued to the GPU when the abort arrived and ran to completion anyway. vLLM's own prompt-token counter records a prompt only once its first output token appears, so it missed a third to half of all computed prompt tokens, and a monitor or autoscaler reading it would have seen much less work than the engine was doing.

What restored service was stopping retries from creating new work. A retry budget at the proxy in front of vLLM, which refused retries beyond 10 percent of first attempts, brought all three models back but delivered about as many of the burst's calls as doing nothing, roughly half on Qwen2.5-7B. Request coalescing, in which the proxy attaches a retry to the execution its call already has running so the retry receives that output, delivered every call from the burst onward on all three models, some of them late. On Qwen2.5-7B the two ideas collided: when cancellation aborted the running execution immediately, retries had nothing to attach to and service did not return, while waiting 10 seconds before cancelling, longer than the client's pause before retrying, let retries attach again and service recovered; in those runs the proxy never actually had to cancel anything.

The authors conclude that where every call must be answered, as when an agent cannot finish its task without it, a proxy should attach retries to live executions and keep them alive long enough for the retry to arrive, while an application that would rather lose a call than get it late may prefer shedding retries with cancellation. Coalescing requires every attempt of a call to carry one identifier; the OpenAI SDK they tested sent none of its own, though it resent a header the caller added. All of this comes from one vLLM version on one GPU type, with synthetic single-turn prompts and one retry policy, and the collapse appeared only when the burst's backlog approached the client's deadline. By the authors' arithmetic, a client that waits 600 seconds would need a backlog of about ten minutes before the same loop could start.