What is temperature in an LLM?
Erwin Berkouwer
What is temperature in an LLM?
Temperature is a parameter that determines how 'cautious' or how 'bold' a language model (LLM) is when choosing the next word in its response. A low temperature produces predictable, consistent answers; a high temperature produces more variation and creativity, but also a higher chance of less coherent output.
Think of temperature as a dial between 'always say the most likely word' (low) and 'give less obvious words a chance too' (high).
The value typically ranges from 0 to 2, where 0 makes the model nearly fully deterministic and values above 1 produce increasingly random and less coherent answers.
How does temperature work technically?
A language model calculates a probability distribution over all possible words that could come next, for every word it generates. Without any temperature adjustment, the model would consistently (or almost consistently) pick the highest-probability word, which can lead to repetitive and sometimes dull output.
Temperature rescales this probability distribution before a word is chosen:
- Low temperature (e.g. 0 to 0.3): the distribution becomes sharper, the model almost always picks the most likely word. Result: consistent, predictable answers.
- Moderate temperature (e.g. 0.5 to 0.8): a balance between consistency and variation, suitable for most general-purpose use.
- High temperature (e.g. 1 and above): the distribution flattens, so less likely words also get a real chance. Result: more variation, but also more risk of incoherent or inaccurate output.
Important to know: temperature does not change the model's underlying knowledge or facts. It only affects how the model chooses between words it already 'knows'.
Why does this matter for SMEs?
If you use AI in your business, for example for customer communication, content creation, or data analysis, the temperature setting directly determines how reliable and consistent the output is. Some use cases call for predictability, others benefit from variation.
| Use case | Recommended temperature | Why |
|---|---|---|
| Customer service responses | Low (0 to 0.3) | Consistency and reliability come first |
| Data extraction or classification | Low (0 to 0.2) | You want the same answer for the same input |
| Content ideas or brainstorming | High (0.7 to 1) | Variation and originality are desired |
| Writing marketing copy | Moderate (0.5 to 0.8) | Balance between brand consistency and fresh phrasing |
When building AI agents for business processes, temperature is one of the first settings we tune, because a poorly set temperature can make an agent unpredictable, or conversely too rigid. Especially in automated workflows that run without supervision, every deviation adds up, so a deliberate temperature choice prevents surprises later in the process.
Example
Suppose you ask a model twice: "Write an opening line for a newsletter about our new product."
At temperature 0.1 you'll likely get nearly the same sentence twice. At temperature 0.9 you'll get two clearly different, possibly more creative sentences, but the risk of an awkward or ill-fitting phrasing also increases.
When to adjust it, when not to
Adjust the temperature when:
- You notice answers are too repetitive or predictable for a creative task
- An automated process shows too much variation where consistency is needed
- You're building an agent that must deliver structured output (e.g. JSON or fixed formats), in which case temperature should stay low
Don't adjust the temperature if the output is already good, or if the actual problem lies elsewhere, for example in the prompt itself. A poor prompt is not fixed by raising the temperature.
Related terms
Temperature is often mentioned alongside other sampling parameters like 'top-p' (nucleus sampling) and 'top-k', which influence word choice in a similar but distinct way. Together, these settings shape a model's behaviour alongside the prompt itself.
Want to know how these settings play out in practice for your business processes? Our AI consultancy helps configure and test models for specific use cases, and the free AI scan maps where AI delivers the most value for you.
Frequently asked questions
What's a good default temperature?
For most business use cases, a value between 0.3 and 0.7 is a safe starting point, depending on whether you want more consistency or more variation.
Can temperature cause a model to make up facts?
A high temperature increases the chance of incoherent or less accurate output, but hallucination (making up facts) also occurs at low temperature and has multiple causes.
Is temperature the same across every AI model?
The principle is similar, but the exact scale and effect can differ per model and provider.
Do I need to set the temperature myself?
If you use an off-the-shelf chatbot, the temperature is usually already fixed. When building your own AI applications via an API, it is a setting you configure yourself.
Frequently asked questions
Short, clear answers so you can decide faster.
What's a good default temperature?
For most business use cases, a value between 0.3 and 0.7 is a safe starting point, depending on whether you want more consistency or more variation.
Can temperature cause a model to make up facts?
A high temperature increases the chance of incoherent or less accurate output, but hallucination also occurs at low temperature and has multiple causes.
Is temperature the same across every AI model?
The principle is similar, but the exact scale and effect can differ per model and provider.
Do I need to set the temperature myself?
If you use an off-the-shelf chatbot, the temperature is usually already fixed. When building your own AI applications via an API, it is a setting you configure yourself.
