Why Evaluate: Demo Quality vs Product Quality

The demo vs product gap

A common story: A team builds a RAG prototype on internal company documents. It looks amazing. Questions like:

• "What is our return policy?"
• "How do I reset my password?"
• "What's the office address in Austin?"

All return perfect answers. The team pitches to leadership, gets approval for the product launch. A week after shipping, user questions flood in:

• "Why does the bot give me conflicting information about returns in different regions?"
• "It keeps saying I can reset my password, but the link is broken."
• "The address it gave is outdated; we moved offices last month."

What happened? The team tested on 5 hand-picked questions, all of which their documents answered correctly. But production traffic includes:

• Edge cases (regional differences, exceptions, outdated information)
• Unanswerable questions (features not documented, recent changes not yet reflected)
• Adversarial inputs (people trying to break the bot)
• Rare but important questions (accessibility, HIPAA compliance, fraud prevention)

None of which appeared in the 5 demo questions.

This is the demo vs product gap: A system that works on carefully curated examples can fail badly on real, messy, diverse user traffic.

Level 0: Vibes-based evaluation

What it is: Someone tries a few prompts in the playground and decides "It feels good."

Examples:

• "I asked it three things, and they all looked right to me."
• "The answers flow naturally and sound authoritative."
• "Compared to the previous version, it seems better."

Failure modes:

• Selection bias (you're testing the easiest questions).
• Confirmation bias (you're unconsciously testing only the cases you expect to work).
• No measurement (you can't compare two versions objectively).
• No consistency (tomorrow's evaluation might contradict today's).

Cost: Zero dollars. Effort: ~30 minutes.

Outcome: A system that passes human intuition but fails in production.

Level 1: Spot checks

What it is: A recurring manual review of ~10-20 test questions, usually without a formal scoring rubric.

Examples:

• "Every Friday, the PM tests 15 questions and logs whether the answers look right."
• "Before shipping a prompt change, we check 20 examples from the test set."

Improvement over level 0:

• At least you're documenting which questions you tested.
• You can compare two versions side-by-side.
• Testing is periodic, not one-off.

Failure modes:

• No formal scoring (is 17/20 good? Is it an improvement from 15/20?).
• No coverage strategy (you're not ensuring the test set includes edge cases).
• Reviewer fatigue (human judgment degrades on the 100th question).
• No versioning (you can't reproduce last month's test results).

Cost: ~1-2 hours per week for someone to do manual testing.

Outcome: Better than level 0, but still no objective threshold for shipping. "The answers look good" can mean different things to different reviewers.

Level 2: Golden set + metrics

What it is: A versioned test set with defined, objective metrics. You run the same set every time you change anything in the pipeline.

Components:

• Golden set: 50-200 representative questions (easy, hard, edge cases), each with labeled ground truth (expected sources, acceptable answers).
• Metrics: For each question, score retrieval (Did we find the right chunks?) and generation (Given those chunks, is the answer good?).
• Reproducibility: The set and metrics are versioned like code; every change to the pipeline is measured.

Example (very simplified):

Test case 1:
  Query: "What is the refund window?"
  Ground truth sources: ["policy/returns.md:section-2", "faq/refunds.md"]
  Acceptable answers: ["30 days", "within 30 days", "one month"]
  Metrics:
    - Retrieval recall: Did we find policy/returns.md? (yes/no)
    - Answer correctness: Does the answer mention 30 days? (yes/no)

Test case 2:
  Query: "How much does the Pro plan cost?"
  Ground truth sources: ["pricing.md:pro-tier"]
  Acceptable answers: ["$99/month", "ninety-nine dollars per month"]
  Metrics:
    - Retrieval recall: (yes/no)
    - Answer correctness: (yes/no)

Run this test set on your system, calculate aggregate metrics:

• Retrieval recall: 85% (we found the right chunk in 85 out of 100 questions)
• Answer correctness: 78% (we got an acceptable answer in 78 out of 100)

Improvement over level 1:

• Objective scores (you can definitively say "78% is better than 73%").
• Coverage strategy (you ensure the test set includes diverse question types).
• Regression detection (if next week's score drops to 71%, you know something broke).

Failure modes:

• The golden set doesn't cover production distribution (you tested easy cases but 30% of real traffic is edge cases).
• Metrics don't match what users actually care about (you optimize for retrieval recall but users care about answer completeness).
• Over-tuning to the test set (the system memorizes it or exploits quirks).

Cost:

• Setup: ~2-4 weeks (curating 100+ test cases, defining metrics).
• Maintenance: ~2-4 hours per week (running tests, adding new cases).

Outcome: You can now make objective release decisions: "We only ship if retrieval recall >= 85%."

Level 3: Automated CI + calibration

What it is: Metrics run on every code change (not just periodically), and scores are calibrated against human labels.

Setup:

• Every pull request that changes the prompt, model, chunking, or retrieval logic automatically runs the golden set.
• If the metrics drop below thresholds, the PR can't merge without manual review.
• On a sample of the golden set (e.g., 20% of cases), human reviewers label the correct answer. Automated metrics are compared to human labels.

Example:

PR #456: "Improve chunking strategy"
  [CI trigger: Run golden set]

  Results:
    - Retrieval recall: 87% (was 85%) ✓
    - Answer correctness: 79% (was 78%) ✓
    - MRR (Mean Reciprocal Rank): 0.71 (new metric)

  [Manual calibration on 20 cases]
  Disagreements: 2 out of 20
    - Case 7: Metric said "correct", human said "partially correct"
    - Case 19: Metric said "incorrect", human said "correct but incomplete"

  Calibration result: Metrics are 90% in agreement with humans.
  Decision: Approve merge.

Improvement over level 2:

• Continuous validation (every change is tested, not just weekly).
• Metric calibration (you know your automated metrics correlate with human judgment).
• Scale without hiring (automated testing lets you ship faster).

Failure modes:

• Humans and LLM judges disagree on what "correct" means (e.g., is a partially correct answer good enough?).
• Metric gaming (your system learns to exploit the metric without actually improving).

Cost:

• Setup: ~6-8 weeks (build CI pipeline, define metrics, calibrate).
• Maintenance: ~5-8 hours per week (reviewing disagreements, updating rubrics).

Outcome: You can ship changes with confidence. "This change passed CI and is calibrated to human judgment."

Level 4: Production monitoring

What it is: Live traffic is continuously sampled and scored. Quality metrics drift triggers alerts and rollbacks.

Setup:

• ~1-5% of user queries are evaluated in real-time using automated metrics or human review.
• Key metrics are tracked over time (retrieval quality, answer correctness, cost, latency).
• Thresholds are set: "If retrieval recall drops below 80%, page on-call."

Example:

Last 1000 live queries:
  - Retrieval recall: 82% (within expected range)
  - Answer correctness: 76% (↓ 2% from yesterday, investigating)
  - p95 latency: 2.1s (↓ 0.3s from yesterday, good)
  - Cost per query: $0.015 (stable)

Alert triggered: Answer correctness trending down
  [Investigate]
  Root cause: New document batch has poor metadata labels
  Action: Pause ingestion, fix labels, re-index

Improvement over level 3:

• Real, not synthetic data (production users reveal failure modes lab testing misses).
• Drift detection (you catch quality regressions within hours, not after a week).
• Business context (you track cost and latency alongside quality).

Failure modes:

• Logging overhead (sampling 1-5% of traffic at scale requires infrastructure).
• False alarms (temporary blips trigger pages; teams stop trusting alerts).
• Expensive to label (hiring humans to evaluate 50 queries per day is costly).

Cost:

• Setup: ~3-4 months of engineering (logging, dashboards, alerting).
• Maintenance: ~10-15 hours per week (triaging alerts, improving metrics).

Outcome: A production system that catches regressions and continues to improve over time.

Moving up the maturity ladder

From level 0 to level 1: Create a simple spreadsheet with 15-20 test questions and results. Takes a day, immediate payoff.

From level 1 to level 2: Define metrics (retrieval recall, answer correctness, latency) and apply them consistently to a versioned set. Takes 2-3 weeks but is the inflection point where evaluation becomes reproducible.

From level 2 to level 3: Build a CI pipeline that runs metrics on every PR. Integrate human calibration for a sample of cases. Takes 6-8 weeks but transforms evaluation into a continuous practice.

From level 3 to level 4: Add production instrumentation, logging, and alerting. Takes 3-4 months but gives you real, ongoing feedback from actual users.

Most teams building RAG should aim for level 2 before shipping, and level 3 before scaling. Level 4 is for high-traffic, business-critical systems.

Why evaluation matters: A real case study

Company: A fintech startup built a RAG chatbot to answer customer questions about accounts, transactions, and tax forms.

Timeline:

• Week 1: Team builds RAG system, tests on 5 questions, decides it's ready. (Level 0)
• Week 2: Ship to 5% of customers as a beta feature.
• Week 3: Customer support receives 50+ tickets: "The bot said my account is overdrawn but I have a positive balance." (Outdated cached data.) "The bot cited the 2023 tax form instead of 2024." (Incorrect source versioning.)
• Week 4: Emergency rollback. Team investigates. Realizes that their 5 test questions never included:
  • Questions about accounts with multiple concurrent transactions.
  • Questions about recent changes (new features, form updates).
  • Boundary cases (zero balance, frozen accounts).
• Week 5-6: Team builds a proper golden set (100 questions, covering easy, hard, and edge cases) and defines metrics (retrieval accuracy on source documents, answer correctness vs ground truth). Discovers their system only scores 68% on the golden set (but was 100% on the 5 demo questions).
• Week 7-10: Fix retrieval bugs, improve prompts, add caching invalidation.
• Week 11: Re-test on golden set, now at 87%. Careful beta rollout to 10% of customers. Monitor live quality.
• Week 12: No new support tickets related to quality. Ship to 100%.

Cost of level 0: 3 weeks of reputation damage, 50+ support tickets, team morale hit, re-work.

Cost of level 2 (golden set + metrics): 2 weeks of upfront work before shipping.

The team would have caught the issues immediately and shipped a better system the first time.

Before you optimize, evaluate

A common anti-pattern: Teams optimize RAG systems without a baseline.

"We changed our chunking strategy. The answers feel better now."

But better by what metric? Retrieval recall? Answer correctness? Latency? Cost? Compared to the old system, how much better?

Always establish evaluation before optimization:

1. Define your golden set (100-200 representative questions).
2. Measure your baseline system against it.
3. Change one thing (new model, new chunking, new prompt).
4. Measure again.
5. Compare.

Without this, you're flying blind. You might optimize for speed and accidentally hurt accuracy. You might optimize for accuracy and accidentally increase cost 10x.

The business case for evaluation

RAG systems that skip evaluation often seem cheaper in the short term. No time spent on labeling, no infra for testing, ship fast. But production failures are expensive:

• Support tickets for bad answers (labor cost).
• Damage to user trust (harder to rebuild than to build initially).
• Emergency rollbacks and firefighting (engineering time).
• Potential compliance/liability issues for certain domains (financial, medical).

A fintech company that ships an unevaluated RAG system and gets one material question wrong can face regulatory scrutiny. A healthcare company faces liability. A B2B SaaS company loses customers.

Conversely, spending 2-4 weeks upfront on a formal evaluation practice (golden set, metrics, calibration) usually pays back within 1-2 months through prevented failures and faster iteration.

Common mistake

Treating evaluation as a checkbox activity. "We ran a test set once, got a score, shipped it. Done."

Evaluation is continuous. As your corpus grows, as user traffic patterns shift, as edge cases emerge, your golden set and metrics need to evolve. A test set from 6 months ago is probably obsolete now.

Also: Only measuring aggregate metrics. "We're at 81% answer correctness" hides the fact that you're at 95% on simple questions but 40% on complex, multi-part questions. Slice your metrics by question type, difficulty, and domain. Use those slices to guide what to fix next.