Intel / article

Generation configurations: temperature, top-k, top-p, and test time compute

Source: huyenchip.com
Author: Chip Huyen
Date: Published Jan 16, 2024

Context

Generation settings change how a selects its next piece of text. Chip Huyen's sampling guide explains that lower temperature concentrates selection on already likely continuations, while higher temperature gives less likely alternatives more opportunity.

Top-k and top-p instead restrict the candidates available for selection. Top-k retains a fixed number; top-p retains enough of the most likely candidates to reach a chosen cumulative probability. That makes top-p's candidate count vary with the model's uncertainty. These controls explain changes in output variety, but none checks whether the selected statement is true.

Key takeaways · AI-distilled

Temperature works by dividing the logits by T before softmax. With logits [1,3] the model picks B 88% of the time at T=1, 98% at T=0.5, and 73% at T=2. Lower T sharpens toward the top choice; higher T flattens the field.
Temperature 0 is a fiction — you cannot divide a logit by zero. Providers skip the softmax entirely and take the argmax, the highest-logit , every time.
Top-k's real win is compute, not style. Softmax needs two passes over the whole vocabulary; restricting it to the top 50-500 logits turns a 100,000-token pass into a tiny one.
Top-p (nucleus) sizes the candidate set per instead of fixing it: sum probabilities in descending order until they cross p. 'Answer yes or no' collapses to two tokens; an open-ended question keeps hundreds.
Providers hand back logprobs — probabilities on a log scale — because with a 100,000-token vocabulary many raw probabilities are small enough to round to zero. Probabilities that look random are also a sign the model learned little.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
test-time compute — Spending more computation when the model answers — thinking longer, trying multiple attempts — to buy accuracy without training a bigger model.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Key quotes

“Recently, I went over 3 months’ worth of customer support requests of an AI startup I advise and found that ⅕ of the questions are because users don’t understand or don’t know how to work with this probabilistic nature.”

“The higher the temperature, the less likely the model is going to pick the most obvious value (the value with the highest logit), making the model’s outputs more creative but potentially less coherent.”

“The more fickle a model is, the more we can benefit from sampling multiple outputs. The optimal thing to do with a fickle model, however, is to swap it out for another.”

“Some are against constrained sampling because they believe the resources needed for constrained sampling are better invested in training models to become better at following instructions.”

Read the source huyenchip.com