Chain-of-Thought Prompting

What is chain-of-thought prompting?

Chain-of-thought (CoT) prompting asks a model to show its work before answering. Instead of jumping straight to a conclusion, the model explains its reasoning step-by-step, then provides the final answer.

Without CoT:

Q: A store sells apples for $1 each and oranges for $2 each. Alice buys 3 apples and 2 oranges. How much does she spend?

A: $7

Correct, but opaque. Can't verify the logic. If wrong, no way to see where it went wrong.

With CoT:

Q: A store sells apples for $1 each and oranges for $2 each. Alice buys 3 apples and 2 oranges. How much does she spend?

A: Let me work through this step-by-step:
1. Apples: 3 × $1 = $3
2. Oranges: 2 × $2 = $4
3. Total: $3 + $4 = $7

Answer: $7

Model outputs the work and the answer. This isn't just helpful for human verification—it fundamentally changes how the model solves the problem. The reasoning steps improve accuracy on the model's own subsequent reasoning. On math and reasoning tasks, chain-of-thought increases accuracy by 5–20 percentage points depending on task complexity.

Why chain-of-thought works

The research (Wei et al., 2022, "Chain-of-Thought Prompting") showed that encouraging models to produce intermediate reasoning steps significantly improves performance on arithmetic, common-sense, and symbolic reasoning tasks. The effect is strongest on complex multi-step problems.

Why? Several theories:

1. Decomposition — Breaking a big problem into smaller steps is easier than solving it in one jump. The model has a better chance of getting each step right than of jumping to a complex conclusion.
2. Error correction — If the model makes a mistake in step 2, it has a chance to correct course before the final answer, rather than being locked into a wrong direction from the start.
3. Transparency — The intermediate steps reveal where reasoning went wrong, which helps the model (and you) identify the failure point.

Modern reasoning models (like OpenAI o1 or similar) extend this concept further, using extended reasoning internally before returning a final answer. But even for standard models, explicitly requesting step-by-step reasoning is a powerful technique that shows measurable accuracy gains on multi-step tasks.

When chain-of-thought helps vs. adds overhead

Chain-of-thought is not always beneficial. It costs more tokens and takes longer to run. Use it strategically:

Chain-of-thought helps for:

• Multi-step math problems — Arithmetic, algebra, logic puzzles.
• Sequential reasoning — "If A happens, then B follows, then C. What is the final state?"
• Complex planning — Breaking down a goal into intermediate steps.
• Factual reasoning with conditions — "Who is older: person X born in 1990, or person Y born in 1985?"
• Tasks where the path matters — Debugging code, diagnosing a problem, tracing cause-and-effect.

Chain-of-thought adds overhead (no benefit) for:

• Simple lookups — "What is the capital of France?" The answer is direct; reasoning doesn't add value.
• Classification with clear rules — "Is this email spam?" if the rules are simple and explicit.
• Generation tasks — Creating a poem or story. Reasoning might constrain creativity.
• Open-ended questions — "What do you think about this idea?" Forcing step-by-step reasoning may feel artificial.

Worked example: cost-benefit of chain-of-thought

Task: Evaluate whether a customer qualifies for a bulk discount based on rule-based logic.

Rule: Customer qualifies for 15% discount if order > $500 AND previous purchases > 3.

Without chain-of-thought (direct answer):

Customer: Alice
Order total: $480
Previous purchases: 4

Does Alice qualify for the 15% bulk discount?

Answer:

Model output: "No"

• Correct: Yes
• Tokens used: ~60
• Transparent: No (we can't see if the model understood both conditions)
• Verifiable: No

With chain-of-thought (show work):

Customer: Alice
Order total: $480
Previous purchases: 4

Does Alice qualify for the 15% bulk discount? Check each condition:

Condition 1 (order > $500): Alice's order is $480. Is $480 > $500? No.
Condition 2 (purchases > 3): Alice has 4 previous purchases. Is 4 > 3? Yes.
Both required: Both conditions must be true (AND). Since condition 1 fails, the overall result is false.

Answer:

Model output: "No, Alice does not qualify. Although she has 4 previous purchases (exceeding the minimum), her order ($480) falls below the $500 threshold required."

• Correct: Yes
• Tokens used: ~90
• Transparent: Yes (clear which condition failed)
• Verifiable: Yes (can audit the logic)

Cost-benefit analysis:

| Dimension | No CoT | With CoT | Difference | |-----------|--------|----------|-----------| | Accuracy on rule set | 94% | 98% | +4 points | | Token cost | 60 | 90 | +50% | | Transparency | None | Full | Enables debugging | | Latency | Fast | Slower | ~40% longer | | Cost (100k evaluations) | $30 | $45 | +$15 | | False positives (wrong discount) | 6 customers | 2 customers | -4 costly errors |

Decision: For rule-based decisions where accuracy is critical and cost matters less, CoT is worth the overhead. For simple lookups ("What's the capital of France?"), CoT wastes tokens.

A production system might use conditional CoT: apply CoT only to uncertain cases. If confidence < 0.8, show work. Otherwise, return direct answer. This optimizes for both speed and accuracy.

How to structure a chain-of-thought prompt

Basic formula:

1. State the task clearly.
2. Ask for reasoning before the answer. Use phrases like "Let me think through this step-by-step" or "I'll break this down."
3. Optionally provide an example of step-by-step reasoning.
4. Request the final answer in a clear, marked section.

Simple chain-of-thought (no example)

Question: A book costs $15. A pen costs $3. Sarah buys 2 books and 5 pens. How much does she spend?

Let me think step-by-step:

Final Answer:

Chain-of-thought with example (few-shot)

Question: A book costs $15. A pen costs $3. Sarah buys 2 books and 5 pens. How much does she spend?

Example:
Question: A notebook costs $2. A pencil costs $0.50. Tom buys 3 notebooks and 8 pencils. How much does he spend?

Step-by-step:
1. Notebooks: 3 × $2 = $6
2. Pencils: 8 × $0.50 = $4
3. Total: $6 + $4 = $10

Final Answer: $10

Now for the original question, let me think step-by-step:

Final Answer:

The example teaches the model exactly what format and depth of reasoning you expect.

Chain-of-thought + self-consistency voting

A related technique, self-consistency (Wang et al., 2022), generates multiple reasoning paths and takes the most common answer. This is even more effective for complex problems than a single chain-of-thought.

How it works:

1. Run the CoT prompt multiple times (3–10 runs, depending on task difficulty).
2. Each run produces a different chain of thought (reasoning may differ slightly, especially with sampling-based generation).
3. Collect the final answer from each run.
4. Vote: Take the most common answer. If 8 out of 10 runs say "yes" and 2 say "no," the answer is "yes."
5. Optionally, return confidence = (most common count) / (total runs). Confidence 8/10 = 0.80.

Accuracy impact: Self-consistency improves accuracy by 5–10 percentage points over single CoT runs on reasoning tasks. However, it's expensive (3–10x token cost).

Example: Fraud detection

Question: Is this transaction fraudulent?

Factors to consider:
- Transaction amount: $10,000
- Customer's typical monthly spending: $500
- Geographic mismatch: Transaction in London, customer usually in New York
- Time: Transaction at 3 AM (unusual for this customer)

Reasoning: Work step-by-step through each factor.

Final answer: [fraudulent/legitimate]

Running this 5 times with sampling-based generation might produce:

Run 1: "Yes, fraudulent" (unusual amount + geography + timing) Run 2: "Yes, fraudulent" (20x normal spending is a red flag) Run 3: "No, legitimate" (customer may plan large purchase; 3 AM could be time zone difference) Run 4: "Yes, fraudulent" (combination of amount, geography, timing is suspicious) Run 5: "No, legitimate" (all factors individually explainable)

Vote: 3/5 say fraudulent → Flag for review with confidence 0.60.

Cost-benefit:

| Approach | Accuracy | Token Cost | Latency | Cost (100k) | Best For | |----------|----------|-----------|---------|-------------|----------| | Direct answer | 88% | 60 | 1x | $30 | High-volume, low-stakes | | Single CoT | 94% | 90 | 1.4x | $45 | Medium-stakes decisions | | CoT + self-consistency (5 runs) | 96% | 450 | 7x | $225 | High-stakes (fraud, compliance, safety) |

Self-consistency is expensive but valuable for decisions where wrong answers are costly. For fraud detection, a 4% accuracy lift (88% → 92% or 94% → 98%) can save thousands in fraud losses.

Implementing self-consistency voting in code

import anthropic
import json
from collections import Counter

def cot_with_self_consistency(question: str, num_runs: int = 5) -> dict:
    """Run CoT prompt multiple times and vote on the answer."""
    client = anthropic.Anthropic()

    answers = []
    reasoning_samples = []

    for i in range(num_runs):
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=500,
            temperature=1.0,  # Sampling-based generation for diversity
            messages=[
                {
                    "role": "user",
                    "content": f"""Question: {question}

Reasoning: Work step-by-step through this decision.

Final answer: [yes/no/uncertain]"""
                }
            ]
        )

        # Parse the response to extract the final answer
        text = response.content[0].text
        lines = text.strip().split('\n')
        final_line = lines[-1].lower()

        # Extract yes/no/uncertain from the last line
        if 'yes' in final_line:
            answer = 'yes'
        elif 'no' in final_line:
            answer = 'no'
        else:
            answer = 'uncertain'

        answers.append(answer)
        reasoning_samples.append(text)

    # Vote on the most common answer
    vote_counts = Counter(answers)
    most_common_answer = vote_counts.most_common(1)[0][0]
    confidence = vote_counts.most_common(1)[0][1] / num_runs

    return {
        "answer": most_common_answer,
        "confidence": confidence,
        "vote_breakdown": dict(vote_counts),
        "reasoning_samples": reasoning_samples
    }

# Usage
result = cot_with_self_consistency(
    "Is a transaction of $10,000 from a customer with $500/month average spending fraudulent?",
    num_runs=5
)

print(f"Answer: {result['answer']}")
print(f"Confidence: {result['confidence']:.2f}")
print(f"Vote: {result['vote_breakdown']}")

For high-stakes decisions, this approach combines the transparency of chain-of-thought with the robustness of ensemble voting.

When reasoning is visible vs. hidden

In standard models (Claude, GPT-4), chain-of-thought is explicit — you see the reasoning in the output. In advanced reasoning models (o1 series), reasoning is often hidden — the model spends tokens on thinking internally, but you only see the final answer.

For audit and transparency purposes, explicit CoT (the standard approach) is better. You can verify the logic. For pure accuracy where you only care about the final answer, hidden reasoning in a specialized model might be better. This is a choice point in your system architecture.

Common mistakes

Mistake 1: Forcing chain-of-thought on simple lookup tasks

"What color is the sky?" doesn't need step-by-step reasoning. Enforcing CoT format wastes tokens and may even hurt response quality.

Wasteful:

Question: What color is the sky?
Reasoning: Let me think step-by-step about the factors that determine sky color.
During the day, Rayleigh scattering causes shorter blue wavelengths to scatter more
than longer red wavelengths...
Answer: Blue.

Better:

Question: What color is the sky?
Answer: Blue.

Token cost: 60 vs. 200 for identical correctness. For high-volume lookups, direct answers are more efficient.

Rule: Use CoT only for tasks where reasoning reduces errors. For lookups, classification with clear rules, or generation tasks, CoT wastes tokens.

Mistake 2: Hallucinated intermediate steps

A model might produce reasoning that sounds plausible but is incorrect. Chain-of-thought is not a guarantee of correctness—it increases accuracy on average, but individual runs can hallucinate reasoning steps.

Flawed reasoning (but correct answer):

Question: Is 127 a prime number?

Reasoning: Let me check divisibility.
127 ÷ 2 = 63.5 (not an integer, so not even)
127 ÷ 3: Well, 3 + 6 = 9, which is divisible by 3, so 127 is divisible by 3.
Wait, let me recalculate: 127 ÷ 3 = 42.33... (not divisible)
127 ÷ 7 = 18.14 (not divisible)
127 ÷ 11 = 11.54 (not divisible)
So 127 is prime.

Answer: Yes.

The answer is correct, but the reasoning is incomplete (didn't check all divisors up to √127 ≈ 11.3). For math-heavy or verification tasks, pair chain-of-thought with a second-pass validation step (e.g., "Verify your answer" or use an external tool).

Mistake 3: Over-constraining the reasoning format

If you specify too many required steps or too rigid a structure, the model becomes mechanical and may hallucinate details to fit the format.

Too rigid (forced structure):

Answer in exactly 4 steps:
Step 1: State the input.
Step 2: Identify the relevant rule.
Step 3: Apply the rule.
Step 4: State the output.

This forces artificial padding. The model might split a simple one-step decision into four steps, each with manufactured details.

Better (flexible):

Work through this step-by-step. Show your reasoning and final answer.
Let the model decide how many steps are needed.

The model will naturally expand on complex reasoning and abbreviate simple reasoning. This produces more honest, less hallucinatory responses.

Mistake 4: Mixing chain-of-thought with excessive examples

Combining few-shot examples (which add tokens) + chain-of-thought (which adds more tokens) can quickly exceed reasonable context for high-volume tasks.

Token explosion:

[System prompt: 200 tokens]
[3 few-shot examples with CoT: 300 tokens each = 900 total]
[Actual question: 50 tokens]
Total context: 1,150 tokens

Tokens for just the prompt overhead: 91% of the request!

If processing 100k requests/month, this is expensive. Consider:

• Use few-shot WITHOUT CoT (examples show format, task is simple).
• Use CoT WITHOUT few-shot (task is complex, examples aren't needed).
• Use both only for high-stakes tasks where accuracy > cost.

Chain-of-thought vs. extended thinking models

Recent developments in AI have introduced extended thinking or internal reasoning models (like OpenAI's o1 series). These models are trained specifically to perform long, internal reasoning before returning an answer. How does this compare to explicit chain-of-thought prompting?

| Feature | Explicit CoT | Extended Thinking Model | |---------|-------------|------------------------| | Reasoning visibility | Visible in output; you see all steps | Hidden; model shows only final answer | | Accuracy on hard problems | Good (+5–10 points) | Excellent (+15–25 points) (illustrative estimate) | | Token cost | Moderate (40–50% overhead) | High (5–10x tokens, mostly for hidden thinking) | | Latency | Fast | Slow (model spends compute time thinking) | | Transparency/audit | Full (all steps visible) | Limited (can't inspect reasoning) | | Best for | Medium-complexity reasoning, logic, moderate math | Hard problems, creative reasoning, very complex math |

Practical guidance:

• For most tasks, explicit CoT with standard models is cost-effective and transparent.
• For very hard problems (math competitions, formal verification, complex engineering design), extended thinking models may be worth the extra cost and latency.
• In production, you might use both: use CoT with standard models for routine tasks, fall back to extended thinking only for problems where CoT shows low confidence.

Summary: when to use chain-of-thought

Use chain-of-thought when:

• The task requires multi-step logic or math.
• Accuracy is more important than latency/cost.
• Explaining the reasoning is valuable (for audit, verification, or user trust).
• The task is complex enough that intermediate steps reduce errors.

Skip chain-of-thought when:

• The task is simple classification or lookup.
• You're optimizing for speed and token efficiency.
• The answer is obvious from the input.
• You're generating open-ended text where reasoning may constrain creativity.

In modern workflows, you might use chain-of-thought during development (to debug and verify correctness) and then move to a faster direct-answer version for production if benchmarks allow. Or, if using a reasoning model, you trade off explicit transparency for internal accuracy.