Thinking Machines opens a research blog, starting with inference nondeterminism
- Source
- x.com
- Date
Today Thinking Machines Lab is launching our research blog, Connectionism. Our first blog post is “Defeating Nondeterminism in LLM Inference” We believe that science is better when shared. Connectionism will cover topics as varied as our research is: from kernel numerics to prompt engineering. Here we share what we are working on and connect with the research community frequently and openly. The name Connectionism is a throwback to an earlier era of AI; it was the name of the subfield in the 1980s that studied neural networks and their similarity to biological brains.
Context
Turning off random word selection does not necessarily make an AI service repeatable. The first research article from Thinking Machines' Connectionism blog explains how changing the number of requests processed together can change the order of arithmetic operations. Computer rounding then produces small numerical differences that can change the model's next word choice. The analysis traces this to calculations that vary with batch size.
The team implemented calculations that preserve their arithmetic order across batches. In its reported repeated-prompt experiment, all 1,000 responses then matched. This provides a concrete route toward reproducible model tests: control the serving calculations as well as the sampling settings. Implementation and experiments.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- determinism — Whether the same input reliably produces the same output — something LLM systems mostly lack, which changes how you test and debug them.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Checking sign-in…
Loading comments…