Zero-Shot vs. Few-Shot Prompting

Zero-Shot vs. Few-Shot: The Trade-Off

Zero-shot prompting means asking the model to do a task with only instructions, no examples. You say "Classify this email as positive or negative sentiment" and the model responds.

Few-shot prompting means including a small number of worked examples (typically 1–5) in your prompt so the model can see the pattern before it tackles your actual task.

Neither is always better. The choice depends on three factors:

1. Task clarity — Can you describe it in 2-3 sentences such that a person would understand?
2. Format sensitivity — Does the exact output structure matter?
3. Cost tolerance — How much is a 2-3x token increase worth?

Recent research (Su et al., 2023) shows that examples are not just about format—they help the model understand the task's underlying distribution. A model with examples produces outputs that are statistically closer to the expected output distribution than a model without examples, even when phrasing alone would suggest they should be identical.

When zero-shot is enough

Zero-shot works for tasks where:

1. The task is obvious from language. "Extract the main topic from this document." Most models understand "topic" and "document."
2. The expected output is simple. A yes/no answer, a single word, or a short phrase.
3. There's little ambiguity. The boundary between categories (or the expected format) is clear from the instructions alone.
4. Speed and cost matter. You're processing thousands of requests and a few percentage-point improvements in accuracy aren't worth the added token cost.

Example zero-shot task (works well):

Classify the following review as either positive or negative.

Review: The product arrived on time and works as advertised. Highly recommend.

Classification:

Most models will output "positive" reliably. No examples needed. The instructions are clear enough.

A practical test: if you can describe the task in 2-3 sentences such that a person would instantly understand it, zero-shot is usually sufficient. If the task requires explaining multiple categories, nuances, or edge cases to a person, few-shot is worth considering.

When few-shot helps

Few-shot prompting improves consistency when:

1. The task requires a specific format. You want JSON, a table, a bulleted list, or a specific sentence structure.
2. Ambiguity exists. A borderline case might be positive or negative depending on context. An example shows which interpretation you prefer.
3. Subtle patterns matter. You care whether a response is "polite but firm" vs. "apologetic" vs. "defensive." Examples clarify the tone.
4. Edge cases exist. You need to show the model how to handle incomplete data, contradictions, or unusual inputs.

Example zero-shot task (would struggle):

Classify the following review as positive, negative, or mixed. Output as JSON.

Review: The product is great, but shipping was slow. Would buy again though.

Output:

Without examples, you might get plain text, or JSON with different field names, or inconsistent formatting. Few-shot is safer.

Same task with few-shot examples:

Classify the following review as positive, negative, or mixed. Output as JSON with the structure shown in the examples.

Example 1:
Review: The product arrived on time and works as advertised. Highly recommend.
Output: {"classification": "positive", "confidence": 0.95}

Example 2:
Review: Broken on arrival. Terrible customer service. Never again.
Output: {"classification": "negative", "confidence": 0.98}

Example 3:
Review: Good quality, but expensive. Not sure it's worth the price.
Output: {"classification": "mixed", "confidence": 0.72}

Now classify this review:
Review: The product is great, but shipping was slow. Would buy again though.
Output:

With examples, the model knows exactly what structure to return and how to interpret "mixed" (in this case, positive overall despite a complaint). Consistency improves significantly.

How many examples is enough?

The diminishing returns typically kick in around 3–5 examples. Here's what the research and practice suggest:

• 1 example: Better than zero-shot for format. Helps the model understand the expected output structure.
• 2–3 examples: Usually the sweet spot. Enough to show a pattern without doubling your token count.
• 4–5 examples: Helps with edge cases and complex tasks. Some improvement over 3, but less than the jump from zero to one.
• 6+ examples: Rarely worth it. You're mostly paying more tokens with little additional benefit. The model has usually learned the pattern by 3–5.

The exact cutoff depends on task complexity. A simple yes/no classification might only need 1 example (or zero). A complex reasoning task might genuinely benefit from 4–5.

Worked example: sentiment analysis scaling

Zero-shot baseline:

Classify the sentiment of this customer review as positive, negative, or neutral.

Review: The app crashes when I open the settings menu.

Sentiment:

Result: "negative" — correct, but confidence unknown. Occasionally returns "negative (frustrated)" or other variations. One out of ten times, it returns "mixed" or "critical" even though the category was never offered. The model is guessing.

One example (few-shot):

Classify the sentiment of this customer review as positive, negative, or neutral.

Example:
Review: The app is fast and intuitive. Love the dark mode.
Sentiment: positive

Review: The app crashes when I open the settings menu.

Sentiment:

Result: More consistently "negative." Format is predictable. Nine out of ten times, the model returns exactly one of the three categories. Much more reliable.

Three examples (few-shot):

Classify the sentiment of this customer review as positive, negative, or neutral.

Example 1:
Review: The app is fast and intuitive. Love the dark mode.
Sentiment: positive

Example 2:
Review: The app crashes when I open the settings menu.
Sentiment: negative

Example 3:
Review: The app works, but I wish it had dark mode.
Sentiment: neutral

Review: Battery drain is worse than my old phone.

Sentiment:

Result: Consistently "negative." The examples clarified that "negative" applies to functional complaints, not just crashes. Also, the addition of a "neutral" example taught the model that lukewarm feedback is distinct from outright criticism. Ten out of ten times, returns one of the three categories with consistent reasoning.

Comparison:

• Zero-shot: Fastest, cheapest. Occasionally wrong on edge cases. Non-standard outputs (hallucinated categories).
• One example: Slightly more reliable. Better format consistency. Tokens: ~150.
• Three examples: Most reliable. Handles edge cases better. Better alignment with expectations. Tokens: ~300–400.

For a high-volume task (thousands of reviews), the extra tokens might not be worth it unless accuracy really matters. For a critical task (moderating user safety reports, customer escalation triage), the extra reliability of three examples is worth the cost.

Real-world trade-off analysis: fintech email classification

Imagine you're building a system to classify customer support emails for a fintech company. Categories: account issue, payment problem, fraud concern, or general inquiry.

Zero-shot baseline:

Classify this customer email into one of: account issue, payment problem, fraud concern, or general inquiry.

Email: [text]

Answer: [category]

Results on a test set of 100 emails:

• Accuracy: 88%
• Token cost: ~60 per request
• API cost per 100k emails/month: $20 (illustrative estimate based on Claude 3.5 Sonnet pricing)
• Misclassifications: 12 emails routed to wrong team
• Fraud reports (correct category): ~87% (3 fraud emails missed out of 23)
• Business impact: Missed fraud reports potentially cost $1000s in fraud losses; misrouted emails cause SLA breaches

Few-shot (3 examples):

You are a support email classifier for a fintech company.

Categories:
- account issue: Password resets, login problems, account access
- payment problem: Charges, refunds, invoice disputes
- fraud concern: Unauthorized transactions, suspected compromise, security concerns
- general inquiry: Questions about features, status checks

EXAMPLES:

Example 1:
Email: "I can't log in. I reset my password but still getting 'incorrect credentials' error."
Category: account issue

Example 2:
Email: "I noticed a charge for $150 I didn't make. This looks fraudulent."
Category: fraud concern

Example 3:
Email: "My auto-payment didn't go through yesterday. What's the status?"
Category: payment problem

EMAIL TO CLASSIFY:
[text]

ANSWER: [category]

Results on the same test set:

• Accuracy: 96%
• Token cost: ~200 per request
• API cost per 100k emails/month: $60
• Misclassifications: 4 emails
• Fraud reports detected: 96% (22 out of 23 correct)
• Business impact: Fewer fraud reports missed, better SLA compliance, faster response

Trade-off analysis:

| Dimension | Zero-shot | Few-shot (3 ex.) | Difference | |-----------|-----------|-----------------|------------| | Accuracy | 88% | 96% | +8 points | | Token cost/request | 60 | 200 | +233% | | Monthly API cost (100k emails) | $20 | $60 | +$40 | | Correct fraud detections | 87% | 96% | +9 points | | Misrouted emails/month | 1,200 | 400 | -800 | | Estimated fraud loss (missed fraud reports) | $5,000–50,000 | $500–5,000 | Potential savings |

Decision: Few-shot is correct here. The $40/month in API costs is trivial compared to the cost of missed fraud and SLA breaches. The accuracy lift matters because fraud is high-stakes.

Counterexample: If you're classifying product reviews for a recommendation algorithm where 88% vs. 96% accuracy makes minimal difference to user experience, zero-shot is fine. The $40/month savings is meaningful at scale, and the accuracy difference doesn't move the needle.

Accuracy improvement: empirical data by task type

Research (Brown et al., 2020; "Language Models are Few-Shot Learners") showed that the accuracy gain from adding examples varies dramatically by task type.

| Task Type | Zero-shot Baseline | Few-shot (3-5 ex.) | Improvement | Notes | |-----------|------------------|-------------------|------------|-------| | Sentiment classification | 85–90% | 92–96% | +6 points (illustrative estimate) | Format matters; examples help | | Entity extraction | 70–75% | 82–88% | +12 points | High variance; examples reduce confusion | | Multiple-choice QA | 75–80% | 85–92% | +10 points | Examples teach reasoning patterns | | Text summarization | 40–50% (if judged by ROUGE) | 55–65% | +15 points | Examples critical for length/style | | Simple classification (yes/no) | 92–95% | 94–97% | +2 points | Diminishing returns; format-sensitive | | Complex reasoning (math, logic) | 60–70% | 75–85% | +15 points | Examples crucial for multi-step tasks |

Pattern: Structured, format-sensitive, or reasoning-heavy tasks see the largest gains. Simple classification sees minimal gains. This table is an illustrative estimate based on aggregated research; your specific task may differ. Test both zero-shot and few-shot on your evaluation set (Lesson 2) to measure the real improvement for your use case.

Choosing examples strategically

Not all examples are equally helpful. The examples you choose affect whether few-shot actually improves the model's performance.

Good examples:

• Diverse. Represent different parts of the spectrum (positive, negative, neutral; simple and complex; clear-cut and borderline).
• Consistent formatting. Match the exact output format you expect in the final answer.
• Representative. Look like the actual inputs the model will see, not artificially clean versions.
• Simple. Easy to understand. The pattern should be obvious, not hidden in noise.

Bad examples:

• All similar. Three positive reviews and zero negatives teaches the model nothing about the negative case.
• Inconsistent formatting. One example in JSON, another in plain text, a third in a bulleted list.
• Unrealistic. Perfect, well-written examples don't reflect messy real-world input (typos, sarcasm, contradictions).
• Contradictory. Two examples with the same input but different outputs.

We'll dive deeper into designing few-shot examples in the next lesson.

Cost vs. quality trade-off

Few-shot prompting costs more because you're sending more text to the model. Here's a rough comparison (illustrative estimate, not official pricing):

Doubling accuracy from 92% to 95% (3 percentage points) might be huge in high-stakes domains (medical, legal, security). In a chatbot filtering spam comments, it might not matter. Measure against your actual use case.

Measuring the impact

The best way to decide between zero-shot and few-shot for your specific task is to test both empirically. Create your five-to-ten-case evaluation set (from Lesson 2), run it with zero-shot, record the results, then add examples and run again. Measure the improvement in accuracy, latency, token consumption, and cost. Then decide: is the improvement worth the overhead?

Case study: Email classification scale-up decision

A company processes 500,000 customer support emails per month. Should they use zero-shot or few-shot?

Zero-shot:

• Accuracy: 88%
• Token cost: ~60 per email
• Monthly API cost: ~$200
• Misclassified emails (sent to wrong team): ~60,000
• SLA violations from misrouting: estimated 2–3 hours of delay per 100 misclassified emails = ~100 hours/month wasted

Few-shot (3 examples):

• Accuracy: 94%
• Token cost: ~180 per email
• Monthly API cost: ~$600
• Misclassified emails: ~30,000
• SLA violations: ~50 hours/month wasted

Decision: The extra $400/month in API costs prevents ~30,000 misclassifications and ~50 hours of support team delays. Given that customer support time is expensive, the few-shot approach pays for itself many times over.

For many organizations, a 3–5 percentage point improvement in accuracy is worth a 2–3x increase in token usage, because the cost of mistakes (customer support escalations, refund requests, security breaches) is high. For others (e.g., a recommendation engine where 88% vs. 94% doesn't materially affect user experience), token cost is the constraint, and a small accuracy improvement isn't worth it.

Your evaluation set makes this decision data-driven instead of anecdotal.

Common mistake

Adding examples doesn't automatically improve accuracy. If your examples teach the wrong pattern, few-shot can make things worse.

Imagine you're classifying support tickets, and all your examples happen to be from a specific product category. The model learns to over-weight that category for all new inputs. Or imagine your examples all use a particular tone (very formal) but your real-world tickets are casual. The model becomes biased toward formal tickets as legitimate and casual ones as suspicious.

Bad examples (introduces spurious correlation):

Example 1:
Email: "The API documentation is excellent and the integration was seamless."
Category: general inquiry

Example 2:
Email: "Best support team ever! Very professional."
Category: general inquiry

Example 3:
Email: "confused about pricing"
Category: payment problem

Pattern taught: formal, lengthy, positive writing → general inquiry. Casual, brief writing → payment problem. This is spurious bias, not the intended pattern.

Better examples (diverse and unbiased):

Example 1:
Email: "I want to know if you support webhooks."
Category: general inquiry

Example 2:
Email: "My bill doubled. Why?"
Category: payment problem

Example 3:
Email: "How's our payment status?"
Category: payment problem

Covers both formal and casual phrasing. The pattern is category-specific, not tone-specific.

The solution: Validate your examples. Run a few test cases with and without examples. If the examples hurt accuracy, revise them. If they help, keep them. And always include at least one "mixed" or "ambiguous" example so the model knows it's okay to say "I'm not sure."

Also: never add examples just because other people do. If your task is truly simple and zero-shot works, stick with zero-shot. Spending 3x tokens for no accuracy gain is not a best practice; it's waste.