A/B Testing and Regression Testing for LLM Features

Online and offline testing: two complementary approaches

RAG and LLM features need two types of testing before shipping:

Offline testing (regression testing): Run your golden-set metrics on every code/prompt/model change before it reaches users. If metrics drop below thresholds, block the deploy. Fast, deterministic, catches silent regressions.

Online testing (A/B testing): Roll out changes to a small percentage of real users, measure quality and business metrics, check statistical significance. Catches distribution shift and user-experience issues that golden sets might miss. Slower, but definitive.

Most production systems do both. Offline catches obvious regressions. Online catches subtle ones and validates that lab metrics correspond to real-world improvements.

Offline regression testing: the CI/CD gate

Before any change reaches production, it must pass a regression test. The simplest version:

# test_rag_golden_set.py
import pytest
from rag_system import retrieve_and_generate
from golden_set import GOLDEN_SET  # list of (query, expected_chunks, expected_answer)
from metrics import compute_nDCG, compute_faithfulness


def test_retrieval_quality():
    """
    Regression test: ensure retrieval metrics don't drop below baseline.
    """
    baseline_ndcg = 0.82  # from last release

    ndcg_scores = []
    for query, expected_chunks, _ in GOLDEN_SET:
        retrieved_chunks = retrieve_and_generate(query)["retrieved_chunks"]
        ndcg = compute_nDCG(retrieved_chunks, expected_chunks, k=10)
        ndcg_scores.append(ndcg)

    mean_ndcg = sum(ndcg_scores) / len(ndcg_scores)

    # Fail the test if nDCG drops > 2% from baseline
    assert mean_ndcg >= baseline_ndcg * 0.98, \
        f"Retrieval quality regressed: {mean_ndcg:.3f} < {baseline_ndcg * 0.98:.3f}"


def test_answer_faithfulness():
    """
    Regression test: ensure generation doesn't hallucinate more.
    """
    baseline_faithfulness = 0.88

    faithfulness_scores = []
    for query, _, expected_answer in GOLDEN_SET:
        result = retrieve_and_generate(query)
        answer, context = result["answer"], result["context"]
        faith_score = compute_faithfulness(answer, context)
        faithfulness_scores.append(faith_score)

    mean_faith = sum(faithfulness_scores) / len(faithfulness_scores)

    assert mean_faith >= baseline_faithfulness * 0.97, \
        f"Answer quality regressed: {mean_faith:.3f} < {baseline_faithfulness * 0.97:.3f}"


def test_latency_budget():
    """
    Regression test: ensure change doesn't make the system slower.
    """
    baseline_p95_latency_ms = 500

    latencies = []
    for query, _, _ in GOLDEN_SET:
        import time
        start = time.time()
        retrieve_and_generate(query)
        latency_ms = (time.time() - start) * 1000
        latencies.append(latency_ms)

    latencies.sort()
    p95_latency = latencies[int(len(latencies) * 0.95)]

    assert p95_latency <= baseline_p95_latency_ms * 1.1, \
        f"Latency regressed: p95 = {p95_latency:.0f}ms > {baseline_p95_latency_ms * 1.1:.0f}ms"


if __name__ == "__main__":
    pytest.main([__file__, "-v"])

How to deploy this:

1. In your CI/CD (GitHub Actions, GitLab CI, etc.), add this test to your pipeline
2. Every commit/PR runs the test against the golden set
3. Test fails → PR is blocked, developer must fix it
4. Test passes → proceed to online testing

Setting baselines and thresholds:

• Baseline: The metric value of your last production release. Track these in a baselines.json file.
• Threshold: The tolerance for regression (e.g., 2-5% drop is acceptable; >5% blocks deploy).

{
  "retrieval_ndcg_at_10": {
    "value": 0.82,
    "threshold_percent_drop": 2.0,
    "last_updated": "2026-07-15"
  },
  "answer_faithfulness": {
    "value": 0.88,
    "threshold_percent_drop": 3.0,
    "last_updated": "2026-07-15"
  },
  "p95_latency_ms": {
    "value": 500,
    "threshold_percent_increase": 10.0,
    "last_updated": "2026-07-15"
  }
}

Gotcha: golden-set saturation

If your golden set is small (100 examples), it can saturate: all changes pass the test because the set doesn't cover edge cases. Mitigate:

• Grow the golden set over time: Every month, add 20-30 new hard examples found in production.
• Stratify by case type: Ensure your golden set has billing queries, technical queries, edge cases, etc.
• A/B test the golden set itself: Periodically test whether raters agree with the golden-set labels.

Online A/B testing: experimental design

1. Define metrics (quality + business)

Quality metrics (from golden sets or LLM evaluation):

• Answer faithfulness (mean score)
• Answer relevance (mean score)
• Retrieval nDCG@10
• Latency (p95)

Business metrics:

• User satisfaction (thumbs-up/down, NPS)
• Conversation completion (did the user get their answer or escalate?)
• Session length (longer = more back-and-forth = worse)
• Cost per session

Example metric card:

Metric: Answer Faithfulness
Definition: Mean faithfulness score (0-1) of all answers in a session, computed by LLM judge
Primary or Secondary: Primary
Direction: Higher is better
Minimum detectable effect: +3% (0.88 → 0.91)
Sensitivity: Compute over all sessions; split by session length and domain if possible
Success threshold: +2% with p < 0.05, no regression on business metrics

2. Calculate sample size

You can't run a valid A/B test without knowing how long it needs to run.

from scipy.stats import norm

def calculate_sample_size(
    baseline_metric: float,  # e.g., 0.88 (88% faithfulness)
    minimum_detectable_effect: float,  # e.g., 0.03 (3% improvement)
    alpha: float = 0.05,  # Type I error (false positive)
    beta: float = 0.2,  # Type II error (false negative)
    variance: float = 0.15  # estimate from historical data
) -> dict:
    """
    Calculate sample size for an A/B test.

    Args:
        baseline_metric: baseline value
        minimum_detectable_effect: smallest difference worth detecting
        alpha: significance level (typically 0.05)
        beta: power = 1 - beta (typically 0.8, meaning 80% power)
        variance: standard deviation squared (estimate from past data)

    Returns:
        dict with sample sizes for control and treatment
    """

    # Standard formulas for two-sample t-test
    z_alpha = norm.ppf(1 - alpha / 2)  # two-tailed
    z_beta = norm.ppf(1 - beta)

    pooled_variance = variance * 2  # conservative: assume both groups same variance
    effect_size = minimum_detectable_effect

    n_per_group = ((z_alpha + z_beta) ** 2 * pooled_variance) / (effect_size ** 2)

    return {
        "sample_size_per_group": int(n_per_group),
        "total_sample_size": int(n_per_group * 2),
        "days_to_run": None,  # depends on traffic
    }

# Example
sample_size = calculate_sample_size(
    baseline_metric=0.88,  # baseline faithfulness
    minimum_detectable_effect=0.03,  # want to detect 3% improvement
    variance=0.15  # estimated variance from past data
)

print(f"Need {sample_size['sample_size_per_group']} samples per group")
print(f"At 1,000 sessions/day, this is {sample_size['sample_size_per_group'] / 500:.1f} days")

Rule of thumb for LLM tests: To detect a 3% improvement in a metric (e.g., 88% → 91% faithfulness) with 80% power, you need ~1,000-2,000 samples per group. If your system handles 1,000 sessions/day, run for 2-4 days.

3. Set traffic allocation and duration

Common allocation:

• Control (baseline): 90% of traffic (want most users on stable version)
• Treatment (new prompt/model/retrieval): 10% (test the change)

Duration:

• Minimum: long enough to collect required sample size
• Recommended: 1-2 weeks to catch day-of-week and time-of-day effects
• Maximum: don't run longer than necessary; you're keeping users on a potentially worse version

4. Guard against novelty bias

Users often prefer new things just because they're new, not because they're better. This "novelty bias" fades after a few days.

Mitigation:

• Run tests for at least 1 week (longer if you can)
• Monitor metric trends; if Treatment was 5% better on day 1 but only 1% better on day 6, novelty bias is probably inflating the effect
• Use Bayesian methods that explicitly model "diminishing novelty effect"

def analyze_ab_test_with_novelty(
    control_daily_scores: List[float],  # e.g., [0.88, 0.88, 0.87, 0.86, 0.86, ...]
    treatment_daily_scores: List[float],
    days_to_stabilize: int = 3
):
    """
    Check if treatment effect is changing over time (novelty signal).
    """
    # Compare first N days to last N days
    treatment_early = sum(treatment_daily_scores[:days_to_stabilize]) / days_to_stabilize
    treatment_late = sum(treatment_daily_scores[-days_to_stabilize:]) / days_to_stabilize

    novelty_effect = treatment_early - treatment_late

    if novelty_effect > 0.02:  # > 2% drop suggests novelty bias
        print(f"⚠ Novelty bias detected: effect dropped {novelty_effect:.1%} from day 1 to day {len(treatment_daily_scores)}")
        print(f"   Recommend extending test or discounting early-day gains")
    else:
        print(f"✓ Effect is stable across days. Novelty bias is minimal.")

    return novelty_effect

5. Multiple comparisons problem

If you measure 10 metrics and use p < 0.05 for each, you expect ~1 false positive by random chance (5 × 10 = 0.5). This is the "multiple comparisons problem."

Mitigation:

• Designate primary metric: Only p-test the main metric (e.g., answer faithfulness). Secondary metrics are exploratory.
• Bonferroni correction: If you're testing N metrics, use p < 0.05/N for each. (Harsh, but safe.)
• Bayesian framework: Specify prior beliefs about all metrics, then update. Doesn't require p-correction.

def bonferroni_corrected_threshold(n_metrics: int, alpha: float = 0.05) -> float:
    """
    Compute the corrected significance threshold to control Type I error rate.
    """
    return alpha / n_metrics

# Example: testing 5 metrics
n_metrics = 5
corrected_alpha = bonferroni_corrected_threshold(n_metrics)
print(f"With {n_metrics} metrics, use p < {corrected_alpha:.4f} for each (instead of p < 0.05)")
# Output: use p < 0.01 for each metric

Putting it together: a complete A/B test workflow

def run_ab_test_analysis(
    control_results: List[dict],  # [{"faithfulness": 0.88, "relevance": 0.92, ...}, ...]
    treatment_results: List[dict],
    primary_metric: str = "faithfulness"
):
    """
    Analyze A/B test results with statistical rigor.
    """
    from scipy.stats import ttest_ind
    import numpy as np

    # Extract primary metric values
    control_metric = [r[primary_metric] for r in control_results]
    treatment_metric = [r[primary_metric] for r in treatment_results]

    # Compute summary statistics
    control_mean = np.mean(control_metric)
    treatment_mean = np.mean(treatment_metric)
    effect = treatment_mean - control_mean
    effect_pct = effect / control_mean * 100

    # Statistical test
    t_stat, p_value = ttest_ind(control_metric, treatment_metric)

    print(f"\n=== A/B Test Results ===")
    print(f"Primary metric: {primary_metric}")
    print(f"Control:   {control_mean:.4f} (n={len(control_results)})")
    print(f"Treatment: {treatment_mean:.4f} (n={len(treatment_results)})")
    print(f"Effect:    {effect:+.4f} ({effect_pct:+.2f}%)")
    print(f"p-value:   {p_value:.4f}")

    if p_value < 0.05:
        if effect > 0:
            print(f"✓ STATISTICALLY SIGNIFICANT IMPROVEMENT (p < 0.05)")
            print(f"  Recommendation: SHIP this change")
        else:
            print(f"✗ STATISTICALLY SIGNIFICANT REGRESSION (p < 0.05)")
            print(f"  Recommendation: DO NOT SHIP")
    else:
        print(f"⚠ NOT STATISTICALLY SIGNIFICANT (p >= 0.05)")
        print(f"  Recommendation: RUN LONGER or investigate effect size")

    return {
        "primary_metric": primary_metric,
        "control_mean": control_mean,
        "treatment_mean": treatment_mean,
        "effect": effect,
        "effect_pct": effect_pct,
        "p_value": p_value,
        "statistically_significant": p_value < 0.05
    }

Real example: testing a new retrieval strategy

Scenario: Your team implemented hybrid search (keyword + vector). You want to test if it's better than pure vector search.

Offline regression (week 1): Run the golden set:

• Vector search nDCG: 0.81
• Hybrid search nDCG: 0.85 (+4.9%)

Pass golden set? Yes, +4.9% > +2% threshold.

Online A/B test (week 2-3):

Control:   Pure vector search (90% of traffic)
Treatment: Hybrid search (10% of traffic)

Metrics:
- Primary: Answer faithfulness (need to improve or maintain)
- Secondary: Latency (hybrid might be slower), user thumbs-up

Sample size calculation:
- Current faithfulness: 0.88
- Target improvement: +2%
- Variance: 0.14 (estimated)
- Sample size per group: ~1,400

At 1,000 sessions/day: run for 2-3 days per group = 5-6 days total

Results after 6 days:

Control:    n=7,500 sessions, faithfulness=0.881, latency p95=480ms
Treatment:  n=840 sessions,  faithfulness=0.883, latency p95=510ms

Effect: +0.2% faithfulness, +30ms latency
p-value: 0.36 (not significant)

⚠ RESULT: No significant improvement in primary metric.
Latency increased. Recommend NOT shipping without further investigation.

Next steps:
1. Check if golden set and prod metrics diverged (distribution shift?)
2. Profile hybrid search latency; can we optimize?
3. Maybe the improvement only helps certain query types — re-slice data

Common mistake

Confusing statistical significance with practical significance. A 0.001% improvement in faithfulness might be statistically significant with 10,000 samples, but not worth the engineering effort or latency cost to maintain.

Also, running tests too short. A 24-hour test often shows false positives due to novelty bias or time-of-day effects. Run at least 1 week, preferably 2 weeks.

Finally, don't ignore regression on secondary metrics. If Treatment improves faithfulness +2% but increases latency +20%, it's probably not worth shipping. Specify success criteria for all metrics upfront, not just the primary one.