GPT can give different answers because it usually samples from multiple plausible next tokens rather than always choosing the single most likely one. Tokens are pieces of text, such as words or parts of words. The model generates a response one token at a time, and each choice influences what comes next. Even a small early difference can lead to a differently worded—or differently reasoned—answer.
Several factors affect how much the output varies:
- Sampling settings: The temperature setting adjusts how strongly the model favors likely tokens. Lower values generally produce more consistent outputs; higher values allow more variation. Another setting, top-p sampling, limits the pool of tokens considered by cumulative probability.
- The full conversation: Identical visible prompts are not necessarily identical inputs. Conversation history, system instructions, attached documents, or retrieved information can change the response.
- Model and service changes: A model version update or changes to an application's configuration can affect answers even when your wording stays the same.
- Numerical differences: Small differences in GPU calculations or backend execution can sometimes change which token is selected, particularly when candidates have very similar scores.
Does setting temperature to 0 guarantee the same answer? Not necessarily. Where supported, it generally selects the highest-scoring next token instead of sampling, which reduces variation substantially. But it does not eliminate every source of nondeterminism or protect against model and configuration changes.
For more repeatable results, keep the complete input and generation settings fixed, use a fixed model version where available, and set a random seed if the API supports it. A seed may improve reproducibility, but it is not a universal guarantee. For tasks that need stable formatting, provide an explicit output schema; that constrains the structure, not necessarily the content.
Different wording is normal. Conflicting factual claims are a reliability issue: variation does not mean both answers are correct. Check consequential claims against dependable sources, even if the model repeats the same answer consistently.
