Designing Good Few-Shot Examples

What makes an example good or bad?

Few-shot examples are the second most powerful lever in prompting (after task clarity). But a set of poor examples can sabotage your prompt. Bad examples teach the model the wrong pattern, introduce unwanted bias, or contradict your instructions.

An example is good if it's:

1. Diverse — Covers multiple scenarios or edge cases, not just the happy path.
2. Consistent in formatting — Matches exactly the output format you expect.
3. Clear and simple — Easy for the model to extract the pattern without noise.
4. Representative — Looks like your real-world inputs, not artificially clean versions.
5. Free from contradiction — All examples point toward the same interpretation of the task.

An example is bad if it:

1. Lacks diversity — All examples are similar (same sentiment, same topic, same structure).
2. Inconsistent formatting — Different output styles or structures across examples.
3. Contradictory — Two examples with similar input but different output, no explanation.
4. Unrealistic — Perfect English, ideal input structure, absent the messiness of real data.
5. Unintentionally biased — All examples happen to share a demographic trait, topic, or pattern that skews the model's behavior.

The diversity principle

Your few-shot examples should span the range of cases the model will actually encounter. If you're classifying sentiment, don't show only one positive example and ignore the negative case.

Bad (low diversity):

Example 1:
Input: "Best product ever! Highly recommend!"
Output: positive

Example 2:
Input: "Amazing! Just what I needed."
Output: positive

Example 3:
Input: "Five stars, buy it now."
Output: positive

Now classify: "Decent quality, but overpriced."
Output:

The model has learned "positive" but has never seen a negative or neutral example. It may struggle with the actual input.

Good (high diversity):

Example 1:
Input: "Best product ever! Highly recommend!"
Output: positive

Example 2:
Input: "Decent quality, but overpriced."
Output: neutral

Example 3:
Input: "Broke after one week. Total waste of money."
Output: negative

Now classify: "It works, but I expected better."
Output:

Now the model has seen all three categories and can distinguish between them.

The consistency principle

All examples must use the exact same output format and structure. If one example returns JSON and another returns plain text, the model will be confused about what format to use.

Bad (inconsistent format):

Example 1:
Input: "Love this product!"
Output: positive

Example 2:
Input: "Hate this product."
Output: {"sentiment": "negative"}

Example 3:
Input: "It's okay."
Output: neutral, confidence score: 0.6

Now classify: "Not bad."
Output:

The model has seen three different output formats. It won't know which one to use.

Good (consistent format):

Example 1:
Input: "Love this product!"
Output: {"sentiment": "positive", "confidence": 0.95}

Example 2:
Input: "Hate this product."
Output: {"sentiment": "negative", "confidence": 0.98}

Example 3:
Input: "It's okay."
Output: {"sentiment": "neutral", "confidence": 0.65}

Now classify: "Not bad."
Output:

Every example uses the same JSON structure. The model knows exactly what to return.

The complexity gradient: simple to complex

Order your examples from simplest to most complex. Start with clear-cut cases, then gradually introduce ambiguity and edge cases. This helps the model learn the base pattern before tackling tricky cases.

Bad (no clear order):

Example 1:
Input: "I'm not sure if I like it. It has good features but also annoying limitations."
Output: neutral

Example 2:
Input: "Love it!"
Output: positive

Example 3:
Input: "Waste of money."
Output: negative

Jumping from a complex "mixed" example to simple ones is disorienting.

Good (simple to complex):

Example 1:
Input: "Love it!"
Output: positive

Example 2:
Input: "Waste of money."
Output: negative

Example 3:
Input: "I'm not sure if I like it. It has good features but also annoying limitations."
Output: neutral

Simple examples first, complex one last. The model learns the basic categories before seeing edge cases.

Worked example: rewriting a bad few-shot set

Let's say you're building a prompt to extract and classify issues from customer feedback.

Original (bad) few-shot set:

Extract the main issue mentioned and classify it as bug, feature request, or feedback.

Example 1:
Input: "The login page is broken. I can't get past the password screen."
Output: issue: "password reset broken", classification: "bug"

Example 2:
Input: "I really want dark mode. It would be great for late-night coding."
Output: issue: "dark mode", classification: "feature request"

Example 3:
Input: "You should add dark mode. Also, I love the new dashboard. But the search is slow."
Output: issue: "search performance", classification: "bug"

Now extract the issue from: "It keeps crashing when I try to sync files."
Output:

Problems with this set:

1. Inconsistent output format — Example 1 uses separate lines, Example 2 uses plain text, Example 3 wraps in JSON-like syntax. Inconsistent.
2. Unclear decision logic — Example 3 has multiple issues (dark mode request, praise, slow search). It chose search performance, but why? The instructions don't explain the rule for picking the "main" issue.
3. Missing edge case — No example showing feedback that's neither a bug nor a request (like Example 2's praise). The model may not know how to handle pure feedback.
4. Ambiguous terminology — "issue" could mean "problem" or "topic." Is feedback an "issue"?

Rewritten (good) few-shot set:

Extract the main issue mentioned and classify it as "bug", "feature request", or "feedback".

Definition:
- bug: Something is broken or not working as expected.
- feature request: User asks for a new capability or improvement.
- feedback: User expresses an opinion (praise or criticism) without asking for a specific change.

If multiple issues are mentioned, pick the PRIMARY one using this priority: bugs > feature requests > feedback.

Example 1 (clear bug):
Input: "The login page is broken. I can't get past the password screen."
Output: {
  "issue": "login page password reset broken",
  "classification": "bug"
}

Example 2 (clear feature request):
Input: "I really want dark mode. It would be great for late-night coding."
Output: {
  "issue": "add dark mode",
  "classification": "feature request"
}

Example 3 (feedback only):
Input: "I love the new dashboard. The UI is really intuitive."
Output: {
  "issue": "positive feedback on dashboard UI",
  "classification": "feedback"
}

Example 4 (mixed: request + bug, pick primary):
Input: "You should add dark mode. Also, the search crashes when I use special characters."
Output: {
  "issue": "search crashes with special characters",
  "classification": "bug"
}

Now extract the issue from: "It keeps crashing when I try to sync files."
Output:

Improvements:

1. Consistent output format — All examples return the same JSON structure.
2. Clear definitions — Upfront definitions of each category and a tiebreaker rule for multiple issues.
3. All categories covered — Examples show bug, feature request, and feedback, plus a mixed case.
4. Complexity gradient — Examples 1–3 are simple (one issue each), Example 4 is complex (multiple issues, model must choose).

With this revised set, the model has clear rules and realistic examples. Output is predictable.

Avoiding unintended bias

Examples can accidentally teach the model unwanted associations. This is called spurious correlation — the model learns an association that exists in your examples but isn't the true rule. Research (Clark et al., 2019) shows that even sophisticated models can latch onto spurious patterns if examples are imbalanced.

Biased example set (spurious correlation on tone):

Example 1 (positive):
Input: "The API is well-documented and integrates seamlessly with our microservices architecture."
Output: positive

Example 2 (positive):
Input: "Excellent performance benchmarks. Five-star quality."
Output: positive

Example 3 (negative):
Input: "i dont like it"
Output: negative

Example 4 (negative):
Input: "its broken n stuff"
Output: negative

Pattern taught: formal, technical writing → positive; casual, lowercase writing → negative. This is spurious bias, not the intended pattern. When a real user writes "i love the dashboard, works great!" (informal positive), the model might misclassify it as negative based on capitalization and punctuation alone, not sentiment.

Unbiased example set (tone-independent):

Example 1 (positive, informal):
Input: "i love this, works great!"
Output: positive

Example 2 (positive, formal):
Input: "The API is well-documented and integrates seamlessly."
Output: positive

Example 3 (negative, informal):
Input: "it doesnt work, total waste"
Output: negative

Example 4 (negative, formal):
Input: "Poor documentation and unstable performance."
Output: negative

Now the examples show that both formal and informal language can be positive or negative. The model learns sentiment independent of tone. You've avoided spurious bias.

Pre-deployment bias audit checklist

Before deploying a few-shot prompt, systematically audit your examples:

| Bias Type | Check | Red Flag | Fix | |-----------|-------|----------|-----| | Demographic | Do positive examples mention certain groups/ages/backgrounds? | All positive examples mention "team," all negative mention "solo developer" | Balance: include both team and solo examples in each category | | Language/tone | Is formal language grouped with one outcome? | All high-confidence examples are formal; all ambiguous are casual | Ensure both formal and casual examples exist for each category | | Topic | Are certain topics always paired with outcomes? | All "pricing" examples are complaints; all "feature" examples are praise | Distribute topics evenly across categories | | Length | Are longer inputs always one category? | All positive examples are 50+ words; all negative are <20 words | Match example length to real-world distribution | | Case/punctuation | Do uppercase, lowercase, or punctuation correlate with outcomes? | All negative examples have typos; all positive are perfect English | Include typos and formatting variations proportionally |

Simple manual check: Randomly shuffle your examples. Read them without category labels. What patterns jump out? If you notice a pattern that isn't the intended classification rule, your examples are teaching spurious bias.

Example: Detecting bias in a customer feedback classifier

Original biased set:

Example 1 (happy customer):
"Our enterprise team loves your solution. Seamless integration, world-class support. Highly recommend for large organizations."
Output: positive

Example 2 (happy customer):
"Great product for teams. Amazing performance. Fortune 500 companies trust it."
Output: positive

Example 3 (unhappy customer):
"small biz here, product too expensive for us"
Output: negative

Example 4 (unhappy customer):
"startup owner, cant afford this product"
Output: negative

Biased pattern learned: Enterprise/large organizations → positive; small business/startups → negative. This is wrong. Sentiment should be independent of company size.

Revised (unbiased) set:

Example 1 (happy, enterprise):
"Our team of 500 loves your solution. Seamless integration, world-class support."
Output: positive

Example 2 (happy, small business):
"Perfect for our 5-person startup. Easy to use, great value."
Output: positive

Example 3 (unhappy, enterprise):
"Expensive and slow. Not worth it for large teams."
Output: negative

Example 4 (unhappy, small business):
"The pricing is too high and I don't see the value."
Output: negative

Now company size is balanced across categories. Sentiment is learned independent of organization type.

When to add more examples

A common impulse is to throw more examples at the problem: "If 3 examples are good, maybe 8 will be better." Research suggests this is severely diminishing returns territory. Additional examples beyond 4–5 rarely improve accuracy by more than 1–2 percentage points, while token cost grows linearly.

Legitimate reasons to expand beyond 3 examples:

1. Your task genuinely has many subcategories. If classifying 10 different support ticket types (not just 3), provide 1–2 examples per category. Total: 10–20 examples.
2. Real-world inputs are consistently messy. If your actual tickets are full of typos, mixed languages, and unusual formatting, dedicate 2–3 examples to edge cases.
3. Edge cases are frequent and costly. If mishandling a specific edge case is expensive (security error, compliance violation, fraud), add an example showing the right behavior.
4. The task is reasoning-heavy. For multi-step logic or cause-and-effect reasoning, 4–5 examples can help. For simple classification, 2–3 is enough.

Rule of thumb: For most standard tasks, the sweet spot is 2–4 well-chosen examples. Beyond that, you're paying token cost for marginal gains.

Scaling examples with categories: guidance table

How many examples do you need? It depends on your category count and task complexity:

| Number of Categories | Recommended Examples | Total Tokens (est.) | When to use | |----------------------|----------------------|-------------------|-----------| | 2–3 (simple binary/ternary) | 1–2 | 150–300 | Simple classification; zero-shot often works | | 4–6 (moderate) | 2–4 | 300–600 | Most standard tasks; good balance | | 7–10 (complex) | 3–5 | 400–800 | Finance, legal, multi-class problems | | 11+ (very complex) | 1–2 per category | 800–2000+ | Specialized domains (medical coding, fault diagnosis) |

Rule of thumb: Provide at least one example per category. For categories 1–6, the sweet spot is 2–4 examples total (one "easy" case and one "edge case" per major category group). Beyond 6 categories, you may need category-specific examples, but even then, diminishing returns kick in around 10 total examples.

For example, if classifying support tickets into 10 categories, start with 5 examples (hitting the main categories), test on your evaluation set, and only add more if accuracy stalls. Don't start with 10 examples pre-loaded into every request.

Common mistake

Including contradictory or unexplained edge cases. If you show an example with a certain input receiving classification A, and then another example with a nearly identical input receiving classification B, the model will be confused. It will either guess or latch onto a spurious pattern.

Don't do this:

Example 1:
Input: "The product quality is good."
Output: positive

Example 2:
Input: "The product quality is good, but I don't recommend it."
Output: negative

[No explanation of why the same input can have different output]

The model struggles to understand the distinction. It might learn spurious patterns like "the word 'but' flips sentiment" (wrong) or "longer inputs are negative" (wrong).

Do this instead (explain the distinction):

Example 1:
Input: "The product quality is good."
Output: positive

Example 2:
Input: "The product quality is good, but I don't recommend it."
Output: neutral

Explanation: Example 1 is straightforward positive sentiment. Example 2 is mixed—
while quality is praised, the explicit "don't recommend" indicates an overall
neutral or negative stance that overrides the praise.

Or better, codify the rule upfront:

CLASSIFICATION RULE:
If a reviewer explicitly says "don't recommend," "wouldn't buy again," or "not
worth it," classify as negative or neutral REGARDLESS of any preceding praise.
The negation overrides earlier positive statements.

Example 1:
Input: "The product quality is good."
Output: positive

Example 2:
Input: "The product quality is good, but I don't recommend it."
Output: negative

(Now the rule is clear before you see examples.)

Principle: Clear rules + reinforcing examples > confusing examples without explanation. A few well-explained examples beat many contradictory ones. The model learns rules from your task definition; examples reinforce those rules, not replace them.