How it works

The explanation, with examples you can run.

Temperature

Temperature divides the scores before the softmax. Below 1 it exaggerates the gaps, so the top token wins more often; above 1 it shrinks them, so less likely tokens get a real chance. As temperature approaches 0, sampling becomes greedy decoding: always take the single most likely token.

Top-p and top-k

Top-k keeps the k most likely tokens and discards the rest. Top-p (nucleus sampling, from Holtzman et al.) keeps the smallest set of tokens whose probabilities add up to p. Both exist to cut off the long tail of unlikely tokens that produce nonsense when sampled; top-p adapts to how confident the model is at each step. A common recommendation is to adjust temperature or top-p, not both at once.

Run this to see both on a made-up distribution for the word after "The capital of France is". The random generator is seeded so the counts are repeatable.

Your code

const logits = { Paris: 6.0, paris: 3.5, Lyon: 2.0, a: 1.5, the: 1.2, banana: -1 }

function softmax(scores, temperature) {
  const entries = Object.entries(scores).map(([t, s]) => [t, s / temperature])
  const max = Math.max(...entries.map(([, s]) => s))
  const exps = entries.map(([t, s]) => [t, Math.exp(s - max)])
  const sum = exps.reduce((a, [, e]) => a + e, 0)
  return exps.map(([t, e]) => [t, e / sum]).sort((a, b) => b[1] - a[1])
}

function topP(probs, p) {
  const kept = []
  let total = 0
  for (const [t, pr] of probs) {
    kept.push([t, pr])
    total += pr
    if (total >= p) break
  }
  return kept.map(([t, pr]) => [t, pr / total]) // renormalise
}

let seed = 42
const rand = () => (seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0) / 2 ** 32

const fmt = (ps) => ps.map(([t, p]) => `${t} ${(p * 100).toFixed(1)}%`).join(", ")
for (const temp of [0.3, 1, 2]) {
  console.log(`T=${temp}:`, fmt(softmax(logits, temp)))
}
const nucleus = topP(softmax(logits, 1.5), 0.9)
console.log("T=1.5, top-p 0.9 keeps:", fmt(nucleus))
console.log("1000 samples:", JSON.stringify(sample(nucleus, 1000)))

⌘/Ctrl + Enter

OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

Choosing settings

One right answer (classification, extraction, routing, code that must compile): low temperature, or the provider's recommended default, plus structured output.
Many good answers (brainstorming, varied copy, test data): higher temperature, and generate several candidates if you then pick the best.
Defaults are reasonable. Many teams never touch these knobs and get good results by improving the prompt instead.

At the time of writing, some providers have stopped accepting custom temperature or top-p values on their newest models, especially those that reason before answering. Check the API reference for the model you use instead of assuming every knob exists.

Determinism is a property you build, not a setting

Even at temperature 0, providers say results will not be fully deterministic: requests are batched with others on shared hardware, and floating-point arithmetic is not associative, so near-ties between tokens can resolve differently. One flipped token early on changes everything after it.

If you need repeatable behaviour:

Store the output the first time and reuse it (cache by a hash of model, prompt and settings) when the same input must give the same answer.
Constrain the output (fixed labels, a JSON schema) so variation cannot change the meaning.
Write tests and evaluations that check properties ("contains the order id", "label is one of five", "grader score at least 4") rather than exact strings, and run them several times.
Pin the model version where the provider allows it, and re-run your evaluations before switching.

Check yourself

3 questions. Get them all right to finish this topic.

1.You classify support tickets into five fixed labels. Which sampling setup fits best?
2.You set temperature to 0 and send the same prompt twice. The answers differ slightly. What is the most likely explanation?
3.With top-p = 0.9, which tokens can be sampled?

0 of 3 right so far. Get all three to finish this topic.

Sampling: temperature, top-p and determinism