Temperature, Top-p, and Sampling Parameters

What temperature and top-p actually do

When a language model generates text, it doesn't deterministically pick the next word. Instead, at each step, the model produces a probability distribution over all possible next tokens. These parameters control how the model samples from that distribution.

Temperature

Temperature rescales the probability distribution before sampling. Think of it as a dial from rigid to random.

• Temperature = 0 (or very low, e.g., 0.1): The model always picks the highest-probability token. This is nearly deterministic.

  • Distribution becomes very sharp. The most likely next token dominates.
  • Use for: extraction, classification, math, anything where you want consistent, predictable output.

• Temperature = 1 (default): The original probability distribution from the model. Balanced.

  • The model makes its natural probability choices.
  • Use for: most general tasks, when you want both coherence and some variation.

• Temperature > 1 (e.g., 1.5 or 2.0): The distribution flattens. Lower-probability tokens become more likely.

  • The model has more freedom to explore unusual or creative choices.
  • Use for: brainstorming, creative writing, generating diverse ideas, when predictability is undesirable.

Math intuition: If the model's logits (raw scores before probability conversion) for next tokens are [10, 8, 5, 2], dividing by temperature T reshapes them. At T=0.1, they become [100, 80, 50, 20] (much sharper). At T=2, they become [5, 4, 2.5, 1] (much flatter).

Top-p (nucleus sampling)

Top-p (also called nucleus sampling) filters the probability distribution before sampling. Instead of considering all possible next tokens, it only considers the most probable ones that add up to probability mass p.

• Top-p = 1.0: Consider all tokens (no filtering). If temperature is low, you're essentially picking the highest-probability token. If temperature is high, you're sampling from the full distribution.

• Top-p = 0.9: Keep only tokens whose cumulative probability reaches 90%. Ignore the long tail of very unlikely tokens.

• Top-p = 0.1: Keep only tokens whose cumulative probability reaches 10%. Very restrictive, almost deterministic.

Why use top-p? It prevents the model from sampling extremely unlikely tokens that would be nonsensical. A low top-p is like saying, "Pick from the most sensible options, but not from the random noise at the tail of the distribution."

Temperature vs. top-p

You can use them together:

• Low temperature + low top-p: Very deterministic. Predictable output. For extraction, structured tasks.
• Low temperature + high top-p: Deterministic (temperature dominates), but allows occasional variation from the top-p filtering.
• High temperature + low top-p: Creative, but filtered through the most reasonable options. For brainstorming while avoiding nonsense.
• High temperature + high top-p: Maximum variability. For truly open-ended generation (poetry, fiction).

Choosing settings for your task

Deterministic tasks

Use low temperature (0.0–0.5) and moderate to high top-p (0.8–1.0).

Examples:

• Extracting structured data from text (e.g., "Extract the email address and phone number.")
• Classification ("Is this email spam? Answer: yes or no.")
• Math problem solving
• Code generation with specific requirements

Prompt + parameters example:

Task: Extract the email and phone from this text.

Text: "Contact John at [email protected] or call 555-1234."

Email: [extract here]
Phone: [extract here]

With temperature=0.2, top_p=0.9, the model will consistently extract the correct values. Different runs will produce the same answer.

Balanced tasks

Use moderate temperature (0.7–1.0) and moderate to high top-p (0.85–1.0).

Examples:

• Summarization
• Q&A
• Content generation with some variation
• Most chatbot interactions

Prompt + parameters example:

Summarize this article in 2-3 sentences.

Article: [...]

Summary:

With temperature=0.8, top_p=0.95, the model produces coherent summaries with natural variation across runs. You won't get identical summaries, but they'll all be reasonable.

Creative tasks

Use high temperature (1.2–2.0) and high top-p (0.9–1.0).

Examples:

• Brainstorming ideas
• Creative writing
• Story generation
• Generating multiple diverse versions of content

Prompt + parameters example:

Generate 5 unique product slogans for a coffee company that emphasize sustainability.

With temperature=1.5, top_p=1.0, each slogan will be different, unexpected, and exploratory. Run it again, you'll get different results.

Common misconception: prompt wording and sampling are separate

Many people confuse their prompt with their sampling parameters. But they're independent levers.

False: "To make the model more creative, I'll say 'Be creative' in my prompt."

True: To make the model more creative, you increase temperature (and possibly adjust top-p). Saying "Be creative" in the prompt doesn't actually change what the model does; the sampling parameters do.

Similarly:

False: "I'll ask the model to be deterministic by saying 'Answer with only the facts.'"

True: Asking for facts helps (it's a good prompt), but low temperature forces determinism. The combination is strongest.

Example: the same prompt, different parameters

Prompt:

List 3 ways to improve customer service.

With temperature=0.2, top_p=0.9: Output is predictable. Likely the same 3 ideas every run (training-set commonalities dominate).

With temperature=1.0, top_p=1.0: Output varies. Sometimes mentioning "response time," sometimes "empathy training," sometimes "feedback loops."

With temperature=1.8, top_p=1.0: Output is diverse and sometimes unexpected. Might suggest "customers should wear uniforms to identify themselves" (nonsensical but creative-sounding).

Same prompt, vastly different outputs. This is purely the parameters.

Best practices for parameter selection

Start with defaults, then adjust

Most APIs default to temperature=1.0 and top_p=1.0. These are sensible for most tasks. Only adjust if you have a specific reason.

For production systems, prefer low temperature

If your system is serving users (a chatbot, extraction service, etc.), start with temperature=0.5–0.8. You want consistency and reliability. Vary the prompt if you need variation, not the temperature.

Don't mix very high temperature with very low top_p

High temperature + low top_p is contradictory: temperature says "be creative," top_p says "only use the most likely tokens." You'll get incoherent output.

Monitor for hallucination at high temperature

At high temperature, the model is more likely to generate plausible-sounding but false information. Use guardrails (fact-checking, source citations) if you're generating with high temperature.

Test with your actual inputs

Different domains may behave differently. Test extraction with temperature=0.2 and temperature=0.1 on your real data. One might be better for your particular use case.

Worked example: tuning for a content recommendation system

Scenario: You have a prompt that generates personalized product recommendations. You want to balance:

• Consistency: Users should get reasonable, relevant recommendations.
• Variety: Each user shouldn't see identical recommendations across multiple visits.

First try: temperature=1.0, top_p=1.0

Recommendations vary a lot. One user gets "wireless headphones" one day and "bluetooth speaker" the next. Good variety, but might feel random.

Refined: temperature=0.7, top_p=0.9

Recommendations are more stable but still varied. One user might see "wireless headphones," "noise-canceling earbuds," or "portable speaker," but not "completely unrelated product."

Result: Users feel the system understands them (products are in the same category) while noticing variation across visits (not the same recommendation every day).

Table: parameter effects on recommendation diversity

| Temperature | Top-p | Diversity | Relevance | Best for | |-------------|-------|-----------|-----------|----------| | 0.3 | 0.9 | Low (same recs) | Very high | Consistent, trusted system | | 0.7 | 0.9 | Medium | High | Balanced (most production systems) | | 1.0 | 1.0 | High | Medium | User engagement, variety | | 1.5 | 1.0 | Very high | Lower | Brainstorming, discovery |

API examples

OpenAI (GPT-4, GPT-3.5)

import openai

response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Summarize this article in one sentence."}
    ],
    temperature=0.7,
    top_p=0.95
)

Anthropic (Claude)

import anthropic

client = anthropic.Anthropic()

message = client.messages.create(
    model="claude-3-sonnet-20240229",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "Summarize this article in one sentence."}
    ],
    temperature=0.7
)

Note: Anthropic APIs also support top_p (and top_k) for fine-grained control.

Comparing outputs across temperature settings

import anthropic

client = anthropic.Anthropic()

temperatures = [0.2, 0.7, 1.5]
prompt = "Generate 3 ways to improve team productivity."

results = {}

for temp in temperatures:
    message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=500,
        temperature=temp,
        messages=[
            {"role": "user", "content": prompt}
        ]
    )
    results[f"temperature_{temp}"] = message.content[0].text
    print(f"\n=== Temperature {temp} ===")
    print(message.content[0].text)

# At T=0.2: Responses are predictable, focus on common ideas (clear meetings, documentation)
# At T=0.7: Natural variation, some common ideas plus diverse suggestions
# At T=1.5: More creative, unexpected suggestions (some might be unconventional or unusual)

Parameter tuning checklist

Before deploying a prompt with non-default sampling parameters:

• [ ] Tested on representative data (50+ examples)
• [ ] Baseline metrics recorded at default (T=1.0, top_p=1.0)
• [ ] Rationale for custom parameters documented
• [ ] Tested effect on output quality, latency, cost
• [ ] Compared against default parameters (is the change worth it?)
• [ ] Behavior is stable across multiple runs
• [ ] No sudden mode shifts (where output type completely changes)

Common mistakes

Mistake 1: Using high temperature when you need consistency

Bad practice:

# Extracting structured data
response = model.generate(
    prompt="Extract the email: [email protected]",
    temperature=1.5
)

At high temperature, the model might hallucinate, skip the email, or return invalid data. For extraction, use low temperature.

Better:

response = model.generate(
    prompt="Extract the email: [email protected]",
    temperature=0.2
)

Mistake 2: Assuming higher temperature always means more creativity

False assumption: "To get more creative ideas, I'll set temperature=2.0."

Reality: At very high temperature (>1.5), you get more random output, not necessarily more creative. Randomness is not creativity. A temperature of 1.2 with a well-crafted prompt often beats temperature=2.0 with a vague prompt.

Better approach: Use moderate temperature (1.0–1.3) and craft a prompt that guides the model toward interesting ideas. Prompt engineering beats raw randomness.

Mistake 3: Setting top_p too low

Bad practice:

temperature=0.8, top_p=0.1

You're restricting the model to only the top 10% of probable tokens. This is too constrained for most tasks. You'll get repetitive, stilted output.

Better:

temperature=0.8, top_p=0.9

Top-p should typically be 0.8 or higher unless you specifically want extreme restriction.

Mistake 4: Not re-tuning when switching models

Different models have different default behaviors. GPT-4 may behave sensibly at temperature=1.0, but another model might produce hallucinations at the same setting.

When you switch models, re-baseline with temperature=1.0 and top_p=1.0, then adjust based on observed output quality.

Worked mathematical example: How temperature reshapes a distribution

To build intuition, let's trace through a concrete example.

Raw logits (unnormalized scores) from the model for the next token:

- "the" → 8.0
- "a" → 6.5
- "that" → 4.2
- "which" → 2.1

Step 1: Convert to probabilities (softmax at T=1.0): Using standard softmax: P(token) = exp(logit) / sum(exp(all logits))

- "the" → 0.65 (65%)
- "a" → 0.23 (23%)
- "that" → 0.09 (9%)
- "which" → 0.03 (3%)

Step 2: At T=0.3 (low, deterministic): Divide logits by 0.3 before softmax: [8.0/0.3, 6.5/0.3, 4.2/0.3, 2.1/0.3] = [26.7, 21.7, 14.0, 7.0]

Convert to probabilities:

- "the" → 0.82 (82%) ← Much more likely
- "a" → 0.15 (15%)
- "that" → 0.02 (2%)
- "which" → 0.01 (1%) ← Much less likely

The distribution sharpens. "the" dominates. Very low randomness.

Step 3: At T=1.5 (high, creative): Divide logits by 1.5: [8.0/1.5, 6.5/1.5, 4.2/1.5, 2.1/1.5] = [5.3, 4.3, 2.8, 1.4]

Convert to probabilities:

- "the" → 0.39 (39%) ← Less dominant
- "a" → 0.27 (27%) ← More competitive
- "that" → 0.21 (21%) ← Higher chance
- "which" → 0.13 (13%) ← Better odds

The distribution flattens. All tokens have a reasonable chance. Much higher randomness.

Intuition: Temperature is a scaling factor on the logits. Lower T makes peaks sharper (determinism). Higher T makes them flatter (randomness).

Comparison table: Temperature vs. Top-p vs. Top-k

| Parameter | How it works | Best for | Trade-offs | |-----------|------------|----------|-----------| | Temperature | Rescales all token probabilities uniformly | Controlling overall randomness across tasks | Affects all tokens; can't exclude specific low-probability tails | | Top-p (nucleus) | Keeps only tokens that sum to p probability | Balancing coherence with flexibility; prevents tail tokens | Requires tuning p per task; slower (needs cumulative sort) | | Top-k | Keeps only top k tokens by probability | Simple, hard cutoff; good for bounded sampling | Rigid; may remove useful low-probability tokens or keep too many |

Illustrative strategy for production:

• Start with temperature + top-p (standard approach)
• Use temperature to set randomness level (0.7 for balanced, 1.5 for creative)
• Use top-p to prevent nonsense (0.9 typical, 0.99 if you want more freedom)
• Only use top-k if you have specific latency constraints or need a hard bound on vocabulary size

Real-world case study: Tuning sampling for customer support chatbot

A company deployed a customer service bot using LLM responses. Initial parameters: T=1.0, top_p=1.0 (model defaults).

Observed problems:

• Accuracy: 82% (sometimes helpful, sometimes inconsistent)
• Hallucination rate: ~8% (bot confidently gave wrong info)
• User satisfaction: 3.2/5 (customers felt the bot was unreliable)

Hypothesis: High temperature and top-p allow the model to take too many creative guesses.

Experiment 1: Lower temperature, keep top-p

• Parameters: T=0.5, top_p=1.0
• Accuracy: 88%
• Hallucination rate: ~4%
• User satisfaction: 3.8/5
• Improvement: Consistency improved; some users felt responses were repetitive

Experiment 2: Lower temperature and moderate top-p

• Parameters: T=0.6, top_p=0.9
• Accuracy: 89%
• Hallucination rate: ~3%
• User satisfaction: 4.1/5
• Winner — Balances consistency with natural variation

Experiment 3: Even lower temperature (comparison)

• Parameters: T=0.2, top_p=0.9
• Accuracy: 91%
• Hallucination rate: ~1%
• User satisfaction: 3.5/5 ← Dropped because responses felt robotic and always identical

Final recommendation: T=0.6, top_p=0.9 gives the best balance for customer service (accurate, safe, but not monotonous).

Cost impact (illustrative estimate):

• Lower temperature slightly reduces inference time (~2%) because the model "thinks" less about low-probability options
• Cost per request: baseline $0.01 → optimized $0.0098 (negligible)

Sampling parameter troubleshooting guide

| Symptom | Likely cause | Solution | |---------|------------|----------| | Output is repetitive, same every run | Temperature too low | Increase to 0.7-0.9 | | Output is nonsensical, incoherent | Temperature too high (>1.5) | Lower to 1.0-1.3 | | Occasional hallucinations in factual task | Top-p too high or T too high | Lower top-p to 0.85 and T to 0.3-0.5 | | Model refuses to generate diverse ideas despite high T | Prompt might override parameter; check role/instruction clarity | Recheck prompt. Good prompt + high T > high T alone | | Latency is high on creative tasks | High T + high top-p = more computation | Slightly lower T (try 1.2) or top-p (try 0.95) |

Summary

Temperature and top-p are sampling parameters that control output diversity through mathematical transformations of the probability distribution:

• Temperature: Rescales logits uniformly. Low (0.1–0.5) = deterministic; High (1.2–2.0) = creative.
• Top-p (nucleus sampling): Filters to top-p cumulative probability. Common values: 0.8–1.0.

For most production tasks, start with temperature=0.6–0.8 and top_p=0.9. Adjust lower for deterministic tasks, higher for creative tasks. Always test on real data.

Critical insight: Prompt wording and sampling parameters are separate levers. A good prompt reduces hallucination, but low temperature actually prevents randomness. You can't prompt your way to determinism; you need both good prompts AND appropriate sampling parameters.

Real-world balance: The best parameters are task-specific. Customer support benefited from T=0.6; creative writing might need T=1.2. Test on your actual use case and measure what matters: accuracy for factual tasks, user satisfaction for conversational tasks, coherence for generation.