Generation configurations: temperature, top-k, top-p, and test time compute
huyenchip.com- Category
- Other
- Type
- ARTICLE
- Builder
- @chipro
- Added
- Jul 21, 2026
About
ML models are probabilistic. Imagine that you want to know what’s the best cuisine in the world. If you ask someone this question twice, a minute apart, their answers both times should be the same. If you ask a model the same question twice, its answer can change. If the model thinks that Vietnamese cuisine has a 70% chance of being the best cuisine and Italian cuisine has a 30% chance, it’ll answer “Vietnamese” 70% of the time, and “Italian” 30%. This probabilistic nature makes AI great for cre
What it can do
Adjust randomness of model outputs via temperature setting
Temperature value and generation prompt → Model responses with controlled creativity/determinism
Restrict token sampling to the top-k most probable tokens
Top-k value and generation prompt → Generated text sampled from limited token set
Apply nucleus sampling using cumulative probability threshold
Top-p value and generation prompt → Generated text sampled from dynamic probability mass
Sample multiple outputs to improve response quality via test time compute
Prompt and number of samples/inference compute budget → Multiple candidate outputs with improved model performance
Compute probability distribution over vocabulary tokens
Input text sequence → Probability distribution across all possible next tokens
Generate outputs in a specified structured format
Prompt and target format specification → Formatted output (e.g., structured/JSON responses)
Classify inputs by computing class probabilities
Input data (e.g., email text) → Class label with probability scores (e.g., spam vs not spam)
Why it made the leaderboard
It demystifies why LLM outputs are inconsistent and how sampling knobs (temperature, top-k, top-p) plus test-time compute actually shape responses — the practical mental model you need to tune generation for factuality vs. creativity and to implement structured outputs.
Comments (0)
No comments yet
Indexed by a proprietary survey. Corrections welcome.