Defeating Nondeterminism In Llm Inference

Choosing the most likely next word at temperature zero does not always make an AI server repeat itself. Thinking Machines traces one cause to how requests are grouped for processing. In its engineering study, the server's load changes the batch size, and batch size changes the order in which some arithmetic operations add numbers, producing tiny rounding differences that can flip which word is selected. The study develops versions of those operations whose results do not depend on batch size.

In one test of a Qwen model, 1,000 runs of the same prompt at temperature zero produced 80 distinct completions. With the batch-independent operations, all 1,000 were identical. Reproducibility therefore required changes to the serving software, not just generation settings. The cost was speed: in a separate test on a smaller Qwen model, the authors' unoptimized deterministic mode took 55 seconds versus 26 for default vLLM, or 42 with an improved attention kernel, and the study does not claim identical results across different hardware or software versions.