A/B Testing Prompts in Production

Why offline evaluation isn't enough

Before now, you may have tested prompts offline: gathering a representative dataset, running both prompt variants on it, and comparing accuracy metrics. Offline evaluation is fast and cheap. But it has a critical limitation.

Offline evaluation tests your prompt on a static dataset. Production tests your prompt on real users with real expectations, time pressure, and context you might not have predicted.

Examples where offline and online diverge:

• Your offline test shows prompt A is 2% more accurate at extracting dates. But in production, users interpret timestamps differently by region. Prompt A's format confuses European users. Online metric drops.

• Prompt B scores higher on your test set for summarization quality. But real users find it more verbose. They bounce faster. Online engagement metric drops.

• Your new prompt passes all accuracy checks. But it's wordier, so token costs are 30% higher. Profitability metric drops.

Offline evaluation catches obvious bugs. But production introduces variability (user behavior, timing, context) that no test set captures perfectly. That's why you need an A/B test with real traffic.

Setting up a live A/B test

1. Choose your variants

You should have:

• Variant A (control): The current prompt in production. Baseline.
• Variant B (treatment): The new prompt you want to test. Should have one clear improvement over A (better clarity, fewer tokens, better guardrail framing, etc.).

Only change one significant thing per test. If you change both the prompt structure and the few-shot examples, you won't know which caused improvement or regression.

2. Define metrics (offline + online)

Offline metrics (evaluated on a held-out test set before traffic):

• Accuracy, precision, recall (whatever matters for your task)
• Token count / cost per request
• Latency

These are sanity checks. If variant B is significantly worse offline, don't deploy.

Online metrics (measured during live traffic):

• Primary metric: The business-relevant outcome. Examples:

  • For a customer service bot: "customer issue resolution rate"
  • For content recommendations: "click-through rate"
  • For code generation: "compile success rate"

• Guardrail metrics: Metrics you absolutely must not degrade. Examples:

  • Accuracy should not drop below 95% (if current is 97%)
  • User satisfaction should not drop below 4.0 (if current is 4.2)
  • Hallucination rate should not increase

Define guardrails before the test. If variant B breaches a guardrail, you stop and rollback immediately.

3. Traffic allocation and sample size

Traffic split: Typically 50/50 (each variant gets 50% of users) or 95/5 (5% see the new variant, baseline for safety).

Start with 5% to catch catastrophic failures early. Only move to 50/50 after 5% shows no guardrail violations.

Sample size calculation:

You need enough traffic to detect a meaningful difference with statistical confidence. The formula depends on:

• Baseline conversion/accuracy rate (what % of users currently achieve the goal)
• Minimum detectable effect (MDE): The smallest improvement you care about (e.g., 1% increase in success rate)
• Statistical power: Usually 80% (20% chance of missing a real effect)
• Significance level: Usually 0.05 (5% chance of false positive)

Illustrative estimate: If your baseline accuracy is 80%, you want to detect a 2% improvement (to 82%), at 80% power and 0.05 significance, you need approximately 3,900 samples per variant.

With 1,000 requests per day, that's roughly 4 days per variant if traffic is consistent. So 8 days total for a 50/50 split (4 days each).

Use tools like:

• Statsig Sample Size Calculator
• Optimizely's sample size guide

to compute exact numbers for your metrics.

4. Run the test

During the test:

• Log everything: For each request, record: variant (A or B), offline metrics (accuracy), user ID, timestamp, whether the primary metric was achieved.
• Monitor guardrails in real-time: Plot guardrail metrics on a dashboard. If any breach thresholds, stop immediately.
• Do not peek too early: Running statistical tests every day inflates your false-positive rate (multiple comparison problem). Wait for your pre-determined sample size.

5. Analyze and decide

After reaching sample size:

1. Run statistical tests. Compare variant A and B on your primary metric using a t-test (for continuous metrics) or chi-squared test (for binary metrics like accuracy).

   • P-value < 0.05? Statistically significant difference. (But is it the right direction?)
   • Effect size and confidence interval: What's the magnitude of the difference? Is it practically meaningful?

2. Check guardrails. Does variant B breach any guardrails? If yes, stop. Do not roll out.

3. Check secondary metrics. Did B improve on expected metrics but degrade on unexpected ones (cost, latency, user satisfaction)?

4. Make a decision:

   • Roll out B: B is better on primary metric, no guardrail violations, acceptable secondary trade-offs.
   • Keep A: B is not statistically better, or B violates guardrails, or trade-offs are unacceptable.
   • Iterate: B had promise but failed on a specific guardrail. Revise and test again.

Worked example: testing a customer service prompt

Scenario: Your customer service bot uses a prompt to extract customer intent (refund, complaint, feature request, etc.). Current prompt has 85% accuracy.

Variant A (control): Current prompt, ~600 tokens per response.

Variant B (treatment): Simplified prompt with clearer instructions and guardrails, ~520 tokens per response.

Offline evaluation:

• Variant A: 85% accuracy, 600 tokens
• Variant B: 87% accuracy, 520 tokens

Both pass sanity checks. Variant B is better. Proceed to live test.

Metrics:

• Primary: Intent classification accuracy (on a hold-out set of manually labeled requests)
• Guardrails:
  • Accuracy should not drop below 83% (we tolerate 2% regression)
  • Hallucination rate should stay below 1%
  • Cost per request should not increase

Traffic and sample size:

• Current traffic: 5,000 customer requests/day
• Minimum detectable effect: 1% improvement (to 86%)
• Sample size needed: ~2,500 samples per variant
• With 5,000 requests/day split 50/50, that's 1 day per variant. Run test for 2 days to be safe.

Setup:

• Route 50% of traffic to Variant A, 50% to Variant B
• Log: request ID, variant, intent classification, actual intent (from user), tokens used, timestamp

Test results (after 2 days, 10,000 total requests):

| Variant | Accuracy | 95% CI | Tokens/req | Cost impact | |---------|----------|--------|-----------|------------| | A | 85.2% | [84.1%, 86.3%] | 602 | baseline | | B | 86.8% | [85.7%, 87.9%] | 521 | -13% cost |

Statistical test: p-value = 0.02. Statistically significant improvement for B.

Guardrail check: All guardrails pass. No hallucination increase.

Decision: Roll out Variant B to 100%.

Common pitfalls in prompt A/B testing

Pitfall 1: Testing too many variants simultaneously

Bad: Running A vs. B vs. C vs. D at once.

You need much larger sample sizes to distinguish between multiple variants. You also confuse yourself about what caused the effect.

Better: Test A vs. B only. Once B wins or loses, test B vs. C if needed.

Pitfall 2: Stopping the test early when you see promising results

Bad: After 2 days, Variant B is 5% better. Roll it out now!

You haven't reached your pre-determined sample size. Statistical noise might explain the difference. This is peeking; it inflates false positives.

Better: Run until your sample size target, then analyze. If it's still better, roll out.

Pitfall 3: Forgetting to track confounders

Bad: You run a test during a holiday, or when a competitor releases a product. The external event dominates the metric.

Better: Note external events. If possible, run your test during a "normal" period. Or, control for the event in analysis.

Pitfall 4: Optimizing for the wrong metric

Bad: Variant B increases engagement (users click more) but decreases accuracy (users get wrong answers).

You optimized for clicks, not correctness. Users feel misled, churn increases.

Better: Define your primary metric in advance. Is it accuracy? Engagement? Cost? Guardrails protect you from optimizing for the wrong thing.

Pitfall 5: No guardrails

Bad: Variant B beats A on primary metric, so you roll it out. But it increased hallucinations by 10%.

You didn't define hallucination as a guardrail, so you didn't check. Users see misleading output.

Better: Define guardrails for every metric you care about (accuracy, hallucination rate, latency, cost, compliance). Check them before rollout.

Pitfall 6: Insufficient offline validation

Bad: You skip offline testing and go straight to live A/B test with a prompt that's clearly worse (more tokens, less coherent).

You waste production traffic on a variant that fails immediately.

Better: Always run offline evaluation first. Only proceed to live testing if the new variant is competitive or better offline.

Reducing test duration with smaller effects

If you're only trying to detect a small improvement (e.g., 0.5%), you'll need a huge sample size. To make testing practical:

1. Increase traffic: Batch requests or temporarily increase user traffic if possible.
2. Lower your MDE: Are you sure you care about 0.5%? Maybe 1% is acceptable. Larger MDE = smaller sample size.
3. Use a Bayesian approach: Instead of frequentist A/B testing (p-values, fixed sample size), Bayesian methods can stop earlier if the evidence is strong. (More complex, but faster decisions.)
4. Test during high-traffic periods: Peak traffic = faster sample size accumulation.

Variants for production rollout strategies

Once a prompt test passes, you have rollout options:

Immediate full rollout

Roll out to 100% immediately. Fast, but risky if offline and online metrics diverged.

Best for: Low-risk changes (e.g., rephrasing that doesn't change behavior), high confidence.

Staged rollout

Roll out to 5% → 25% → 50% → 100%, monitoring guardrails at each stage.

Advantages: Catch issues early, users don't all see a bad prompt at once. Disadvantages: Slower, more operational overhead.

Gradual rollout with monitoring

Roll out to 100%, but keep variant A's implementation available. If guardrails breach, automatically switch back to A.

Best for: Mission-critical systems where you want fast deployment but automatic safety net.

Integration with your prompt versioning system

If you're using a prompt library (see the later lesson on team prompt libraries), your A/B test workflow is:

1. New prompt variant created in the library, marked as "draft" or "testing."
2. Offline eval passes. Variant marked as "ready for A/B test."
3. A/B test runs in production. Variant marked "test in progress."
4. A/B test completes. Variant marked as "passed production" or "failed" or "needs revision."
5. If passed, variant can be promoted to "production" and made the default.
6. Previous variant marked as "deprecated" (keep in library for reference).

This creates an audit trail: you can see why variant B replaced variant A, when, with what metrics.

Deeper statistical design: Sample size calculator

A practical formula for sample size when comparing two independent groups (A vs. B) on a binary metric (success/fail):

Illustrative calculation:

• Baseline conversion rate (e.g., accuracy): p0 = 0.85
• Desired improvement: delta = 0.02 (to 87%)
• Significance level (alpha): 0.05 (5% false positive rate)
• Power (beta): 0.80 (80% chance to detect real effect)

Using standard sample size calculators (like Evan Miller's or Statsig), you'd need approximately:

n ≈ 2 * [(1.96 + 0.84)^2 * (p0(1-p0) + p1(1-p1))] / (p1 - p0)^2
  ≈ 2 * [(2.80)^2 * (0.85*0.15 + 0.87*0.13)] / (0.02)^2
  ≈ 3,100 samples per variant

In practice:

• At 1,000 requests/day total: ~3 days per variant (6 days total for 50/50 split)
• At 10,000 requests/day total: ~13 hours per variant (1 day total)

This is an illustrative estimate; actual requirements depend on your specific baseline and acceptable effect size.

Sample size calculator example in Python

import math
from scipy import stats

def calculate_sample_size(
    baseline_rate: float,
    effect_size: float,
    alpha: float = 0.05,
    beta: float = 0.20
) -> int:
    """
    Calculate sample size for binary metric A/B test.

    baseline_rate: Current success rate (e.g., 0.85 for 85% accuracy)
    effect_size: Desired improvement (e.g., 0.02 for 2% gain)
    alpha: Significance level (false positive rate, default 5%)
    beta: Type II error rate (default 20%, so power = 80%)
    """
    # Treatment rate after improvement
    treatment_rate = baseline_rate + effect_size

    # Standard normal z-scores
    z_alpha = stats.norm.ppf(1 - alpha / 2)  # Two-tailed
    z_beta = stats.norm.ppf(1 - beta)

    # Sample size formula for proportion test
    p_avg = (baseline_rate + treatment_rate) / 2
    n = (
        (z_alpha + z_beta) ** 2 *
        (baseline_rate * (1 - baseline_rate) + treatment_rate * (1 - treatment_rate))
    ) / (effect_size ** 2)

    return math.ceil(n)

# Example:
# Baseline accuracy: 85%, want to detect 2% improvement
sample_size_per_variant = calculate_sample_size(
    baseline_rate=0.85,
    effect_size=0.02
)
print(f"Need {sample_size_per_variant} samples per variant")
# Output: ~3100 samples per variant

Real case study: E-commerce recommendation prompt A/B test

A retail company ran an A/B test on their product recommendation prompt.

Setup:

• Current prompt (A): Long, detailed instructions (~1,200 tokens)
• New prompt (B): Simplified instructions, same examples (~900 tokens)

Offline evaluation (before live test):

• Both prompts: ~88% accuracy on test set
• But B used 25% fewer tokens (cost consideration)

Live A/B test design:

• 50/50 traffic split
• Duration: 1 week (estimated 50,000 requests total, 25k per variant)
• Primary metric: Click-through rate on recommendations (did users click?), with secondary accuracy check
• Guardrails:
  • Accuracy should not drop below 85%
  • Hallucination rate < 5%
  • Cost per request should not increase

Results (illustrative case study data):

| Metric | Variant A (Control) | Variant B (Treatment) | Delta | p-value | Significant? | |--------|-------------------|----------------------|-------|---------|--------------| | Primary: Click-through rate | 12.3% | 12.8% | +0.5% | 0.08 | Marginal (not quite p<0.05) | | Secondary: Accuracy | 88.1% | 87.9% | -0.2% | 0.45 | No | | Hallucination rate | 2.1% | 2.3% | +0.2% | 0.62 | No | | Cost per request | $0.0145 | $0.0109 | -24.8% | <0.001 | Highly significant |

Decision:

• Primary metric (CTR) shows a trend but not statistically significant (p=0.08)
• Secondary metrics are all acceptable (no guardrail violations)
• Cost improvement is substantial and significant

Business decision: Roll out Variant B to 100% because:

1. No significant accuracy degradation
2. 25% cost savings is material over time (for illustrative estimate: 10M requests/month × $0.00360 savings = ~$36k/month in savings)
3. The CTR trend (though not significant at p<0.05) combined with no downside makes B the safer choice

Comparison: How to handle different test scenarios

| Scenario | Decision | |----------|----------| | B is significantly better on primary, no guardrail violations | Roll out B immediately | | B has no significant primary impact, but saves 20% cost | Roll out B if cost-conscious; consider rolling out | | B is significantly better on primary, but hallucination increases from 2% to 4% | Do NOT roll out. Hallucination is guardrailed at <3%. Investigate why B hallucinated more. Iterate. | | B is neutral on primary, improves latency by 30% | Roll out B if latency is a constraint; consider if other benefits exist | | B is marginally worse (p=0.10) on primary, no other changes | Do NOT roll out. Insufficient evidence of improvement. Only roll out if cost/latency benefits are massive. |

API example: Logging and analyzing A/B test results

import json
from datetime import datetime
import statistics

class ABTestLogger:
    """Log A/B test results for analysis."""

    def __init__(self, test_id: str, log_file: str = "ab_test_log.jsonl"):
        self.test_id = test_id
        self.log_file = log_file

    def log_result(self, user_id: str, variant: str, metrics: dict):
        """Log a single result."""
        entry = {
            "timestamp": datetime.now().isoformat(),
            "test_id": self.test_id,
            "user_id": user_id,
            "variant": variant,
            **metrics  # Flatten metrics: {accuracy, cost, latency, ...}
        }
        with open(self.log_file, 'a') as f:
            f.write(json.dumps(entry) + "\n")

    def analyze(self) -> dict:
        """Analyze test results. Compute summary stats."""
        variant_a = []
        variant_b = []

        with open(self.log_file, 'r') as f:
            for line in f:
                entry = json.loads(line)
                if entry["test_id"] != self.test_id:
                    continue

                if entry["variant"] == "A":
                    variant_a.append(entry)
                else:
                    variant_b.append(entry)

        # Compute stats for a specific metric (e.g., accuracy)
        a_accuracies = [e.get("accuracy", 0) for e in variant_a]
        b_accuracies = [e.get("accuracy", 0) for e in variant_b]

        return {
            "variant_a": {
                "count": len(variant_a),
                "accuracy_mean": statistics.mean(a_accuracies) if a_accuracies else 0,
                "accuracy_stdev": statistics.stdev(a_accuracies) if len(a_accuracies) > 1 else 0,
            },
            "variant_b": {
                "count": len(variant_b),
                "accuracy_mean": statistics.mean(b_accuracies) if b_accuracies else 0,
                "accuracy_stdev": statistics.stdev(b_accuracies) if len(b_accuracies) > 1 else 0,
            }
        }

# Usage:
logger = ABTestLogger(test_id="prompt_v2_test")

# During the test, log each result:
# logger.log_result(user_id="user_123", variant="A", metrics={"accuracy": 0.92, "cost": 0.0145, "latency": 1.2})

# After the test:
results = logger.analyze()
print(f"Variant A accuracy: {results['variant_a']['accuracy_mean']:.3f} ± {results['variant_a']['accuracy_stdev']:.3f}")
print(f"Variant B accuracy: {results['variant_b']['accuracy_mean']:.3f} ± {results['variant_b']['accuracy_stdev']:.3f}")

Statistical significance testing with scipy

from scipy import stats

def evaluate_ab_test(variant_a_successes: int, variant_a_total: int,
                     variant_b_successes: int, variant_b_total: int) -> dict:
    """Run a chi-squared test on binary metrics."""
    # Create contingency table
    contingency = [
        [variant_a_successes, variant_a_total - variant_a_successes],
        [variant_b_successes, variant_b_total - variant_b_successes]
    ]

    chi2, p_value, dof, expected = stats.chi2_contingency(contingency)

    a_rate = variant_a_successes / variant_a_total
    b_rate = variant_b_successes / variant_b_total
    difference = b_rate - a_rate

    return {
        "variant_a_rate": a_rate,
        "variant_b_rate": b_rate,
        "difference": difference,
        "p_value": p_value,
        "significant": p_value < 0.05,
        "chi2": chi2,
        "recommendation": "Roll out B" if p_value < 0.05 and difference > 0 else "Keep A"
    }

# Example: A/B test results
# Variant A: 850 successes out of 1000
# Variant B: 875 successes out of 1000
results = evaluate_ab_test(
    variant_a_successes=850,
    variant_a_total=1000,
    variant_b_successes=875,
    variant_b_total=1000
)

print(f"Variant A: {results['variant_a_rate']:.1%}")
print(f"Variant B: {results['variant_b_rate']:.1%}")
print(f"Difference: {results['difference']:.1%}")
print(f"P-value: {results['p_value']:.4f}")
print(f"Statistically significant: {results['significant']}")
print(f"Recommendation: {results['recommendation']}")

Feature flag implementation for safe rollout

from enum import Enum
import random

class PromptVariant(Enum):
    A = "control"
    B = "treatment"

class PromptFeatureFlag:
    """Safely switch between prompt variants using feature flags."""

    def __init__(self, control_variant: str, test_variant: str):
        self.control = control_variant
        self.test = test_variant
        self.traffic_allocation = {"A": 0.5, "B": 0.5}  # 50/50 split

    def set_traffic_allocation(self, variant_a_pct: float, variant_b_pct: float):
        """Adjust traffic split (for staged rollout)."""
        assert variant_a_pct + variant_b_pct == 1.0
        self.traffic_allocation = {"A": variant_a_pct, "B": variant_b_pct}

    def get_variant(self, user_id: str) -> str:
        """Determine which variant to show user."""
        # Use user_id for deterministic assignment (same user always sees same variant)
        hash_value = hash(user_id) % 100
        if hash_value < (self.traffic_allocation["A"] * 100):
            return self.control
        else:
            return self.test

# Usage:
flag = PromptFeatureFlag(
    control_variant="prompt_v1",
    test_variant="prompt_v2_improved"
)

# Assign variant to user
user_id = "user_12345"
variant = flag.get_variant(user_id)
prompt = get_prompt(variant)  # Load the appropriate prompt

# After A/B test passes, gradually increase traffic to B
# Day 1-3: 50/50
# Day 4-5: 10% B, 90% A (ramp-down control, catch regressions)
# Day 6+: 100% B (full rollout)
flag.set_traffic_allocation(variant_a_pct=0.9, variant_b_pct=0.1)

Common mistakes and how to avoid them

Mistake 1: Confusing statistical significance with practical significance

Bad: Variant B is statistically significantly better (p < 0.05) but only 0.1% more accurate. You roll out.

Statistical significance just means the difference is real, not due to noise. But is 0.1% worth the operational overhead?

Better: Consider effect size and practical impact. A 0.1% improvement in accuracy might not justify maintenance costs. Set a minimum effect size threshold before the test.

Mistake 2: Early stopping (peeking)

Bad: After 2 days, Variant B is 3% better. You stop the test and roll out.

You haven't reached your pre-computed sample size. Statistical noise might explain the difference. This inflates false positives.

Better: Commit to your sample size calculation before starting. Don't check results daily. Wait for the duration you calculated.

Mistake 3: Testing variants that are too similar

Bad: Variant B changes one comma. You run an A/B test expecting to detect a 1% difference.

You'll need millions of samples. Unless you have strong reason to believe a tiny change matters, don't test it.

Better: Only A/B test meaningful changes (e.g., structural changes, new examples, guardrail additions). Estimate effect size before the test.

Mistake 4: Ignoring user segmentation

Bad: Variant B is better overall (87% vs. 85%) but only for English speakers. For non-English speakers, it's worse (80% vs. 82%).

Average masks the truth. You've degraded service for one segment.

Better: If you suspect segmentation (language, geography, user type), run stratified analysis. Report results per segment.

Mistake 5: No rollback plan

Bad: Variant B passes the test. You deploy. A user finds a critical edge case the test missed. Rollback takes 3 hours.

Better: Before deploying, ensure both A and B are live and switchable (via feature flag). Plan instant rollback.

Summary

A/B testing prompts in production is how you move from "I think this prompt is better" to "I know this prompt is better for my users."

Key workflow:

1. Offline validation first — Sanity check both variants on a held-out test set
2. Define metrics — Primary metric (what you optimize for) and guardrails (what you protect)
3. Calculate sample size — Use statistical power calculations; don't guess
4. Run the test — 50/50 or staged traffic; monitor guardrails in real-time
5. Analyze results — Statistical tests, effect sizes, secondary metrics
6. Make a decision — Roll out, iterate, or stick with baseline
7. Plan rollout and rollback — Feature flags, gradual rollout, automatic safety switches

Production A/B tests are the final proof. They catch what offline evaluation misses: real user behavior, edge cases, and context. Combined with offline evaluation and monitoring, they're how you ship prompts with confidence.