Generation configurations: temperature, top-k, top-p, and test time compute
Source
Chip Huyen
Author
Chip Huyen
Date
Key takeaways · AI-distilled
Temperature works by dividing the logits by T before softmax. With logits [1,3] the model picks B 88% of the time at T=1, 98% at T=0.5, and 73% at T=2. Lower T sharpens toward the top choice; higher T flattens the field.
Temperature 0 is a fiction — you cannot divide a logit by zero. Providers skip the softmax entirely and take the argmax, the highest-logit tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →, every time.
Top-k's real win is compute, not style. Softmax needs two passes over the whole vocabulary; restricting it to the top 50-500 logits turns a 100,000-token pass into a tiny one.
Top-p (nucleus) sizes the candidate set per context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → instead of fixing it: sum probabilities in descending order until they cross p. 'Answer yes or no' collapses to two tokens; an open-ended question keeps hundreds.
Providers hand back logprobs — probabilities on a log scale — because with a 100,000-token vocabulary many raw probabilities are small enough to round to zero. Probabilities that look random are also a sign the model learned little.
Terms in this piece · Glossary
test-time compute — Spending more computation when the model answers — thinking longer, trying multiple attempts — to buy accuracy without training a bigger model.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
It demystifies why LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → outputs are inconsistent and how sampling knobs (temperature, top-k, top-p) plus test-time computeSpending more computation when the model answers — thinking longer, trying multiple attempts — to buy accuracy without training a bigger model.Full definition → actually shape responses — the practical mental model you need to tune generation for factuality vs. creativity and to implement structured outputs.
Key quotes
“Recently, I went over 3 months’ worth of customer support requests of an AI startup I advise and found that ⅕ of the questions are because users don’t understand or don’t know how to work with this probabilistic nature.”
“The higher the temperature, the less likely the model is going to pick the most obvious value (the value with the highest logit), making the model’s outputs more creative but potentially less coherent.”
“The more fickle a model is, the more we can benefit from sampling multiple outputs. The optimal thing to do with a fickle model, however, is to swap it out for another.”
“Some are against constrained sampling because they believe the resources needed for constrained sampling are better invested in training models to become better at following instructions.”
