Same Model, Same Prompt. Why a Different Answer?

You ask for a name for a small seaside café. The model suggests Low Tide. You press regenerate. Now it wants to call the place Salt & Steam.

Same question. Same model. Two perfectly usable signs above the door. What changed?

For a café, the variation is welcome. For a script that extracts invoice totals, it is less charming. To understand both cases, it helps to watch the small decision that happens before each new piece of text appears.

Before the answer, there is a choice

An autoregressive language model produces scores for possible next tokens. A token might be a word, part of a word, or punctuation. The decoding system turns those scores into a choice, appends the chosen token, and asks the model for the next set of scores.

Greedy decoding picks the highest-scoring option at each step. Sampling draws from a probability distribution, so an option with a smaller share can still be selected. The same starting distribution can therefore produce different first choices. Hugging Face’s decoding guide shows both mechanisms.

There is no completed paragraph waiting behind the screen. A reply takes shape as these choices accumulate. Even a tiny fork near the beginning can give the rest of the sentence somewhere else to go.

Temperature changes the odds

Consider the unfinished sentence “The sea looked…”. In our small, invented vocabulary, “calm” starts with half the probability mass. “Silver” has a quarter. Three other options share the rest.

Move the temperature control below. Lower values concentrate probability on the leading option; higher values flatten the distribution. The ranking stays the same in this example. What changes is how much of a chance the less likely choices get.

Temperature scales the model’s scores before softmax. For positive temperature T, the probabilities in this demo are proportional to base_probability^(1/T). At T = 1, they return to the original 50%, 25%, 15%, 7% and 3%. The controls use the same calculation each time. See the temperature implementation.

The label “creativity” on some interfaces can be misleading. This control does not check a fact, improve an argument, or decide whether a café name is any good. A low temperature can keep choosing the same bad answer. A high temperature can simply give weak alternatives more room.

Top-p decides who gets a ticket

Temperature reshapes the distribution. Top-p narrows the pool. It keeps the smallest group of highest-probability tokens whose combined probability reaches the chosen threshold, then rescales the remaining probabilities for sampling. This is also called nucleus sampling. The generation reference distinguishes it from top-k, which keeps a fixed number of candidates.

With the numbers above, a threshold of 0.80 keeps three candidates. The first two add up to 0.75, so the third has to come along. Together they carry 0.90. A threshold is allowed to be crossed; it does not slice a token into pieces.

The original nucleus-sampling paper explored how decoding choices affect open-ended writing. Its useful insight is that the candidate pool can expand when many continuations are plausible and contract when a few dominate. That is a mechanism, not a universal recipe for better answers.

One fork changes the next fork

Choose “calm” and the next words may describe a still harbour. Choose “restless” and they may describe water hitting a wall. The model evaluates the next step against the text already chosen, so the second draw does not use an untouched copy of the first distribution.

This is why two replies can wander far apart even if their opening difference seems trivial. “Compare these tools” can become a list of prices in one run and a discussion of privacy in another. The useful question is whether both replies satisfy the request, rather than whether they use identical sentences.

Does a seed make it repeatable?

A seed gives a pseudorandom generator a repeatable starting point. In a controlled setup, that helps you reproduce sampling. It does not freeze the whole machine.

Model files, software versions, hardware and execution details still matter. PyTorch’s reproducibility notes explicitly warn that identical seeds do not guarantee identical results across releases or platforms, including CPU versus GPU.

There is another easy trap: the same visible message may arrive inside a different conversation. A changed system instruction, an earlier reply, or different retrieved documents changes the input. Chat templates also add structure and control tokens around the text you type. The chat-template guide explains why the complete formatted input matters.

A five-minute experiment worth keeping

Take a task you actually care about. For example: extract the date, supplier and total from one short invoice. Write down the expected fields before you start. Save the exact model version, the full input and the generation settings.

Run the unchanged request several times. Check missing fields, invented values and invalid formatting. Then change one setting and repeat. For writing, use a different scorecard: specificity, awkward phrases, repetition and whether the result sounds like something you would publish.

Start from the model author’s recommended settings. If your runtime supports greedy decoding, include it as a baseline; do not assume that every interface interprets “temperature 0” the same way. In Transformers, sampling is controlled by do_sample, while greedy decoding also assumes a single beam.

Keep the examples that expose a weakness. A folder of ten difficult prompts tells you more about your own workload than an afternoon of regenerating a friendly greeting.

For the café, print your three favourite names and put them on the wall. For the invoice script, keep the validation checks. Variation has a place. You get to decide where.