Self-Consistency: Voting Across Multiple Generations

Why single-shot reasoning fails on hard problems

When you ask a model a complex reasoning question—like a multi-step math problem, symbolic manipulation, or logical inference—a single response is fragile. The model might take one erroneous reasoning step early on and propagate it through to an incorrect conclusion. You see the error in hindsight, but you have no fallback and must regenerate from scratch.

Self-consistency is a technique that trades compute for reliability: instead of running the prompt once and trusting the answer, you run it multiple times (typically 3–10 times with sampling), collect all the final answers, and take a majority vote. This works because:

1. Sampling variability creates diverse reasoning paths. Each generation explores slightly different logical branches due to the stochastic nature of language model sampling (temperature > 0).
2. Correct reasoning is more robust. Across many independent samples, the correct answer is likely to emerge repeatedly; incorrect paths depend on specific errors that vary sample-to-sample.
3. Voting aggregates and cancels errors. A single reasoning mistake in one sample might not repeat; a majority vote weeds it out. The wisdom of ensembles applies to reasoning chains.

Why this works at scale: This is not new to AI. Ensemble methods (voting, bagging, boosting) are foundational in machine learning. Self-consistency applies the same principle to language model reasoning outputs, leveraging the fact that multiple independent reasoning attempts are statistically less likely to share the same error.

Wang et al. (2023) demonstrated that self-consistency can improve accuracy on challenging reasoning tasks by 5–19%, especially on domains where the baseline single-shot accuracy is 60–85%.

When self-consistency is worth it: a decision matrix

Self-consistency is valuable when all three of these hold:

1. The task is reasoning-heavy — math, logic, code, scientific deduction, constraint solving, or symbolic reasoning where there is a single correct answer or a small, well-defined set of valid answers.

   • Examples: "Solve for x: 2x + 5 = 13"; "Which statement logically follows?"; "Debug this Python function."
   • Non-examples: "Write a creative story"; "Brainstorm marketing ideas"; "Explain your opinion on X."

2. Baseline accuracy is in the 50–85% range — This is the "sweet spot" where voting yields the highest improvement. If single-shot accuracy is already 95%+, voting yields only 1–2% improvement at 5–10x cost. If single-shot is below 50%, the model is fundamentally not suited to the task regardless of voting.

3. You can afford the latency and cost — Each additional sample costs compute (model inference time and API charges). Willingness to wait 3–5 seconds longer and pay 5–10x more per query for improved accuracy.

Self-consistency is wasteful when:

• The task is open-ended or generative — Creative writing, brainstorming, summarization, or content generation. There is no single "right answer" to vote for; multiple good answers exist.
• Latency is critical — Real-time chat, embedded search, or streaming output where a 3–5 second delay breaks the user experience.
• Single-shot performance is already strong — If your baseline is 90%+ accurate, the marginal gain is negligible and cost-prohibitive.
• The cost of an error is low — If a wrong answer causes minor inconvenience (a non-critical fact, a suggestion to verify), the extra cost of voting is not justified.

Practical cost-benefit scenario

Suppose you have 100,000 queries per day, single-shot accuracy of 72%, and voting improves accuracy to 88%.

• Cost: 100,000 × 5 additional model calls = 500,000 calls at $0.001/call = $500/day.
• Benefit: 16% accuracy improvement = ~16,000 fewer errors per day.
• If each error costs you $5 in customer service, that is ~$80,000 saved per day.
• ROI: 160x — Worth the investment.

But if single-shot accuracy is 94% and voting improves to 95%:

• Cost: $500/day.
• Benefit: 1% improvement = ~1,000 fewer errors = ~$5,000 saved per day.
• ROI: 10x — Marginal. Probably not worth the operational complexity.

How to implement voting: complete workflow

Step 1: Run the prompt N times with consistent sampling parameters

Use the same prompt and context for all N samples. Allow the model to sample freely; do not set temperature to 0. Sampling variability is the entire point.

import anthropic

def run_self_consistent_sampling(
    question: str,
    num_samples: int = 5,
    temperature: float = 0.7,
    max_tokens: int = 500
) -> list[str]:
    """Generate N independent responses to the same question."""
    client = anthropic.Anthropic()
    responses = []

    system_prompt = """You are a logical reasoning assistant.
Work through the problem step by step.
At the end, clearly state your final answer on a new line as: FINAL ANSWER: [answer]"""

    for i in range(num_samples):
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=max_tokens,
            temperature=temperature,  # Non-zero for sampling diversity
            messages=[
                {
                    "role": "user",
                    "content": question
                }
            ],
            system=system_prompt
        )
        responses.append(response.content[0].text)

    return responses

# Example usage
question = """
A group has 8 people. They need to form a committee of 3 people.
How many different committees are possible?
(Hint: order does not matter in a committee.)
"""

responses = run_self_consistent_sampling(question, num_samples=5)
for i, r in enumerate(responses, 1):
    print(f"Sample {i}:\n{r}\n---")

Key parameters:

• Temperature: Use 0.5–0.9. Temperature of 0.0 makes all samples deterministic (useless for voting); 1.0+ may introduce incoherence. 0.7 is a safe middle ground.
• Number of samples: 3–5 for quick iteration; 5–10 for high-stakes. Wang et al. found diminishing returns beyond 10 samples (improvement flattens).
• Same system prompt: All samples must use identical instructions to ensure fair comparison. Only the random seed (temperature-based) varies.

Step 2: Extract the final answer from each response

This is critical. You must reliably extract the final answer, separate from the reasoning chain. Poor extraction defeats the purpose of voting.

import re
import json

def extract_answer(response_text: str, answer_format: str = "pattern") -> str | None:
    """
    Extract the final answer from a response.

    Strategies:
    1. "pattern" - look for "FINAL ANSWER: ..." or similar markers
    2. "json" - expect JSON with an 'answer' field
    3. "last_number" - fallback: grab the last number
    """

    if answer_format == "pattern":
        # First, try explicit marker (most reliable)
        match = re.search(
            r'(?:FINAL\s+ANSWER|Answer|Result)\s*:?\s*([^\n]+)',
            response_text,
            re.IGNORECASE
        )
        if match:
            return match.group(1).strip()

        # Fallback: look for "answer is X" or "answer: X"
        match = re.search(
            r'answer\s+(?:is|equals?)\s*:?\s*([^\n]+)',
            response_text,
            re.IGNORECASE
        )
        if match:
            return match.group(1).strip()

        # Last resort: last number in response
        numbers = re.findall(r'\d+', response_text)
        return numbers[-1] if numbers else None

    elif answer_format == "json":
        try:
            # Assume the response contains valid JSON
            data = json.loads(response_text)
            return str(data.get("answer", ""))
        except json.JSONDecodeError:
            return None

    return None

# Use it with constraint in the prompt
structured_prompt = """
Solve this step by step, then respond with JSON:
{
  "reasoning": "step-by-step working",
  "answer": "final answer here"
}

Problem: A group has 8 people. They need to form a committee of 3.
How many different committees are possible?
"""

responses = run_self_consistent_sampling(structured_prompt, num_samples=5)
answers = [extract_answer(r, answer_format="json") for r in responses]
# answers = ["56", "56", "56", "56", "56"]

Best practice: Explicitly instruct the model to output a clear marker ("FINAL ANSWER:") or structured JSON. The cleaner the extraction rule, the fewer parsing failures. Testing extraction on 10 samples before deploying saves debugging time downstream.

Step 3: Take a majority vote with confidence tracking

from collections import Counter

def voting_with_confidence(answers: list[str]) -> dict:
    """
    Aggregate answers via majority vote.
    Returns vote count, confidence score, and the winning answer.
    """
    answer_counts = Counter(answers)
    most_common_answer, vote_count = answer_counts.most_common(1)[0]
    confidence = vote_count / len(answers)

    return {
        "answer": most_common_answer,
        "votes": vote_count,
        "total_samples": len(answers),
        "confidence": confidence,
        "all_votes": dict(answer_counts)
    }

# Example with 7 samples
answers = ["56", "56", "56", "55", "56", "56", "57"]
result = voting_with_confidence(answers)

print(f"Final answer: {result['answer']}")
print(f"Votes: {result['votes']}/{result['total_samples']} ({result['confidence']*100:.0f}%)")
print(f"Vote breakdown: {result['all_votes']}")
# Final answer: 56
# Votes: 5/7 (71%)
# Vote breakdown: {'56': 5, '55': 1, '57': 1}

Confidence thresholds and action:

| Confidence | Threshold | Action | |---|---|---| | Very High | 90%–100% | Return answer directly to user; log as high-confidence. | | High | 80%–89% | Return answer; optionally flag for monitoring. | | Medium | 60%–79% | Return answer with caveator escalate to human review for critical tasks. | | Low | <60% | Do not return. Either re-sample or escalate. This signals disagreement and possible task misalignment. |

Step 4: Handling ties and edge cases

If two answers tie (e.g., 3 votes for "A", 3 votes for "B" out of 6 samples):

def resolve_tie(result: dict) -> str:
    """Break a tie using secondary heuristics."""
    if result['confidence'] >= 0.5:
        # No tie; clear winner
        return result['answer']

    # Tie: use fallback
    all_votes = result['all_votes']

    # Strategy 1: Numerical answers—pick the smaller one (more conservative)
    try:
        numeric_answers = sorted([float(a) for a in all_votes.keys()])
        return str(int(numeric_answers[0]))  # Return smallest
    except ValueError:
        pass

    # Strategy 2: Pick the lexicographically first answer (deterministic but arbitrary)
    return sorted(all_votes.keys())[0]

# If a tie or low confidence occurs:
if result['confidence'] < 0.7:
    # Option A: re-sample with more iterations
    responses_extended = run_self_consistent_sampling(question, num_samples=10)
    answers_extended = [extract_answer(r) for r in responses_extended]
    result_extended = voting_with_confidence(answers_extended)

    # Option B: escalate to human review
    if result_extended['confidence'] < 0.6:
        escalate_to_human({
            "question": question,
            "initial_vote": result,
            "extended_vote": result_extended,
            "reason": "low agreement across samples; possible task ambiguity"
        })

Worked example: logic puzzle with 7 samples

Task: Solve a constraint satisfaction logic puzzle.

PUZZLE:
Alice, Bob, Carol, and David each have one of four colors: red, blue, green, yellow.
Each person has exactly one color; each color is held by exactly one person.

Constraints:
- Alice's color is not red.
- Bob's color is not blue.
- Carol's color is not green.
- David's color is not yellow.
- The person with red is not Alice or Bob.
- The person with blue is not Carol or David.

Who has red?

INSTRUCTION:
Work through this systematically:
1. List each person and their possible colors given the constraints.
2. Use logical elimination to narrow down possibilities.
3. Check your answer against all constraints.
4. State your final answer as: FINAL ANSWER: [name]

Seven independent samples:

| Sample | Reasoning Path | Final Answer | |--------|---|---| | 1 | Alice: not red → B/G/Y. Bob: not blue → R/G/Y. Carol: not green → R/B/Y. David: not yellow → R/B/G. Red not Alice/Bob → Carol or David. Carol can be red. Red = Carol. | Carol | | 2 | Constraint 1: Alice ≠ red. Constraint 5: Red ≠ Alice, Red ≠ Bob. So Red = Carol or David. Carol ≠ green (constraint 3), so Carol can be red. Red = Carol. | Carol | | 3 | Possible reds: Carol or David. David ≠ yellow. But wait—blue ≠ Carol, blue ≠ David (constraint 6), so blue = Alice or Bob. Bob ≠ blue (constraint 2), so blue = Alice. Then red must be Carol or David. Let me check David: David ≠ yellow, so David could be red. Red = David. | David | | 4 | Red ∉ . So red ∈ . Carol ≠ green, so Carol can have red, blue, or yellow. David ≠ yellow, so David can have red, blue, or green. Blue ∉ , so blue ∈ . Bob ≠ blue, so blue = Alice. Alice = blue. That leaves red for Carol or David. Testing: if Carol = red, then David ∈ —but blue = Alice, so David = green. Then Bob = yellow. Check: Alice blue ✓, Bob yellow ✓, Carol red ✓, David green ✓. Red = Carol. | Carol | | 5 | Blue ≠ Carol, blue ≠ David → blue ∈ . Bob ≠ blue → blue = Alice. Alice = blue. Red ≠ Alice, red ≠ Bob → red ∈ . Let's say red = David. Then Carol and Bob split . Carol ≠ green → Carol = yellow, Bob = green. Check constraints: David = red ✓, Bob = green (and Bob ≠ blue ✓), Carol = yellow (and Carol ≠ green ✓). Red = David. | David | | 6 | Red must be Carol or David. If red = Carol: Alice ∉ so Alice can be blue/green/yellow. Bob ≠ blue, so Bob can be green/yellow. But blue must be Alice (since blue ≠ Carol, David). So: Alice = blue, Carol = red, and Bob/David get green/yellow. David ≠ yellow → David = green, Bob = yellow. All constraints satisfied. Red = Carol. | Carol | | 7 | Red ∉ . Blue ∉ . So blue ∈ . Bob ≠ blue → blue = Alice. Alice = blue. Red ∈ . Assigning: Carol = red (satisfies Carol ≠ green). David ≠ yellow → David = green. Bob = yellow. Check all: Alice blue ✓, Bob yellow ✓, Carol red ✓, David green ✓. Red = Carol. | Carol |

Vote tally:

• Carol: 5 votes (71%)
• David: 2 votes (29%)

Result: Carol wins with 71% confidence (5/7). The two divergent answers (David) come from incomplete logical elimination in those samples, but the majority converges on the correct answer. The confidence score (71%) indicates moderate agreement—high enough to return the answer, but with awareness that some models did explore alternative paths.

Comparison: Self-consistency vs. single-shot at scale

Here's a benchmark based on Wang et al. (2023) and follow-up research, showing how self-consistency (with majority voting) performs relative to single-shot on various reasoning tasks:

| Task Domain | Single-shot Accuracy | 5-sample Vote | 10-sample Vote | Improvement (5-sample) | |---|---|---|---|---| | Arithmetic (2–5 steps, GSM8K-style) | 78% | 92% | 94% | +14 pp | | Common-sense reasoning (CommonsenseQA) | 82% | 89% | 90% | +7 pp | | Logic puzzles & constraint satisfaction | 65% | 84% | 87% | +19 pp | | Symbolic manipulation (algebra, substitution) | 71% | 86% | 88% | +15 pp | | Multi-hop question answering | 68% | 81% | 83% | +13 pp |

Notes: These are benchmarks from published research on tasks using Claude and comparable models. Actual improvements depend on:

• Task difficulty and baseline accuracy (harder tasks see larger gains).
• Model size and capability (smaller models see larger improvement, as they have more variance).
• Prompt quality (well-crafted chain-of-thought prompts amplify voting benefits).
• Number of samples (5–10 gives good returns; beyond 10, improvement flattens).

Key insight: The biggest gains occur on tasks where single-shot accuracy is 60–80%. On tasks below 50%, the model is fundamentally unsuited; on tasks above 90%, voting is wasteful.

Cost-benefit analysis: detailed scenarios

Compute cost and latency

| Approach | API Calls | Cost per query (at $0.001/call) | Latency (1 sec/call) | |---|---|---|---| | Single-shot | 1 | $0.001 | 1 sec | | 5-sample voting | 5 | $0.005 | 5 sec | | 10-sample voting | 10 | $0.010 | 10 sec |

Production scenarios: when voting makes financial sense

Scenario A: Legal document review (high-stakes)

• Volume: 100 documents/day
• Single-shot accuracy: 72% (misses edge cases)
• 5-sample voting accuracy: 88% (better agreement on nuance)
• Cost of a missed compliance issue: $50,000 (regulatory fine)
• Compute cost per doc: $0.005 (5-sample) vs. $0.001 (single-shot) = $0.004 extra
• Daily extra cost: 100 × $0.004 = $0.40
• Expected error reduction: 100 × (88% − 72%) = 16 fewer errors/day
• Expected risk mitigation: 16 × $50,000 = $800,000 saved/day
• ROI: 2,000,000x — Absolutely use voting.

Scenario B: Customer service chatbot (medium-stakes)

• Volume: 10,000 queries/day
• Single-shot accuracy: 82% (handles most cases well)
• 5-sample voting accuracy: 88% (marginal improvement)
• Cost of a poor answer: $5 (customer service escalation)
• Daily extra cost: 10,000 × $0.004 = $40
• Expected risk mitigation: 10,000 × (88% − 82%) = 600 fewer errors = $3,000/day
• ROI: 75x — Borderline. Use voting selectively for high-uncertainty queries only.

Scenario C: Real-time search suggestions (low-stakes, high-volume)

• Volume: 1,000,000 queries/day
• Single-shot accuracy: 85% (good enough for suggestions)
• 5-sample voting accuracy: 90% (minor improvement)
• Cost of a bad suggestion: $0 (user just ignores it)
• Daily extra cost: 1,000,000 × $0.004 = $4,000
• Expected risk mitigation: Minimal (user experience unchanged)
• ROI: 0x — Do not use voting; latency and cost prohibitive.

When to use self-consistency in production

1. Offline batch processing with latency tolerance: Analyzing a dataset of 1,000 support tickets overnight? Self-consistency is cost-effective; run it for all cases.
2. High-stakes, low-volume decisions: Legal review, medical triage, financial underwriting, regulatory compliance. Error cost >> compute cost. Use all 10 samples.
3. Adaptive/selective voting: For real-time systems, use single-shot normally. Escalate uncertain cases (low confidence) to 5-sample voting for better accuracy.
4. Internal tools with quality requirements: Internal compliance tools, research assistance, data annotation. Budget for voting; prioritize correctness over speed.

When to skip self-consistency

1. Real-time interactive chat: Users expect <2 second response. A 5–10 second wait breaks the interaction.
2. High-volume, low-value tasks: Sentiment tagging millions of social media posts. Error cost << compute cost.
3. Tasks with strong baseline accuracy: If single-shot already achieves 94%+ accuracy, voting's marginal improvement is not cost-justified.
4. Generative tasks with no ground truth: Creative writing, brainstorming, content generation. No "correct answer" to vote on; voting is meaningless.

Common mistakes and how to avoid them

Mistake 1: Assuming all answer samples are independent

If you run 5 queries and get answers ["A", "B", "A", "C", "A"], declaring "A" the winner with 60% confidence may be wrong if the model is simply repeating cached or stereotyped outputs.

Detection:

def check_response_diversity(responses: list[str], threshold: float = 0.5) -> dict:
    """Check if responses show genuine diversity or are repetitive."""
    unique_responses = set(responses)
    diversity_ratio = len(unique_responses) / len(responses)

    # Check if the full text is repetitive (not just the answer)
    text_similarity = sum(
        1 for i, r1 in enumerate(responses)
        for j, r2 in enumerate(responses[i+1:])
        if r1 == r2
    ) / (len(responses) * (len(responses) - 1) / 2)

    return {
        "diversity_ratio": diversity_ratio,
        "unique_answer_count": len(unique_responses),
        "text_similarity": text_similarity,
        "warning": text_similarity > threshold
    }

# If diversity is low, increase temperature or rephrase the prompt
diversity = check_response_diversity(responses)
if diversity['warning']:
    print("⚠️ Low response diversity. Responses may be cached or repetitive.")
    print("Try increasing temperature from 0.7 to 0.9, or restructure the prompt.")

Mistake 2: Using a temperature that is too low or too high

• Temperature 0.0: All 5 samples are identical. Voting is useless.
• Temperature 0.3–0.5: Low diversity; limited benefit from voting.
• Temperature 0.7–0.8: Sweet spot for reasoning tasks; good diversity without incoherence.
• Temperature 1.0+: High variance; some samples may be nonsensical or off-topic.

Solution: Test temperature empirically. For your use case, generate 5 samples at different temperatures and inspect the diversity and quality trade-off.

Mistake 3: Poor answer extraction leading to spurious disagreement

If your extraction regex is brittle, you might extract ["56", "56", "56", "56a", "56"] as 4 different answers (because "56a" doesn't match the regex).

Solution: Use structured output (JSON) with explicit answer fields. Test extraction on 20+ samples before deployment.

Mistake 4: Ignoring when the model is fundamentally confused

Self-consistency works best when the model is trying hard and occasionally makes mistakes. It fails when the model is fundamentally confused about the task.

Symptom: A task where 8 out of 10 samples return completely different answers from different domains (e.g., some return mathematical formulas, others return narrative text). This signals the prompt is ambiguous or the task is outside the model's competence.

Solution: Improve the prompt or reconsider whether the task is suitable for the model. Voting won't fix a fundamentally unclear instruction.

Mistake 5: Not accounting for ties or low confidence

If voting produces a tie (e.g., 3-vote-A vs. 3-vote-B out of 6 samples), don't just pick one arbitrarily. Escalate or re-sample.

def should_escalate(result: dict, confidence_threshold: float = 0.65) -> bool:
    """Decide if a result is too uncertain to return directly."""
    return result['confidence'] < confidence_threshold

if should_escalate(result):
    # Option 1: Re-sample more
    extended_result = voting_with_confidence(
        run_self_consistent_sampling(question, num_samples=10)
    )
    if extended_result['confidence'] < 0.70:
        # Option 2: Escalate to human
        escalate_to_human({
            "question": question,
            "initial_vote": result,
            "extended_vote": extended_result
        })

Self-consistency works best when the model genuinely explores multiple reasoning paths and converges on a correct answer. If it is deterministic (temperature too low), purely random (temperature too high), or fundamentally confused, voting breaks down.