Building a Golden Evaluation Set

Mine from reality, not imagination

A golden set built at a desk by guessing what users might ask is almost always wrong. The questions you invent are the ones you expect to be hard. But production users ask about things you never thought of.

Mine from real sources:

1. Support logs: What are users actually asking support? Extract those questions.
2. Search logs: If you have an older search system or FAQ search, what did users search for?
3. Beta program: Run RAG on a small subset of real users, log every query.
4. Competitor research: What questions do competitors' bots field? What are users asking in their forums?
5. User interviews: Sit with 5-10 users and ask them the questions they want answered.

Example: Mining support logs

Your support team has answered 500 tickets in the past month. Extract the user's original question from each:

Ticket #1001: User asks "How do I reset my password?"
Ticket #1002: User asks "Is my data encrypted?"
Ticket #1003: User asks "Can I use this on Android?"
Ticket #1004: User asks "How do I export my data in CSV?"
...

You now have 500 real queries. Don't use all of them (too many), but use them to build your golden set. If 10% of support tickets ask about password resets, your golden set should include ~10% password-reset questions.

Curating for coverage

Not all 500 mined questions are equally valuable. Some are duplicates. Some are outside the RAG system's scope (e.g., "Can you integrate with Salesforce?" → That's a feature request, not a question the docs answer).

Define tiers:

Tier 1: Core / High-frequency questions

• "How do I reset my password?" (appears in 5% of all tickets)
• "What is your pricing?" (appears in 8% of tickets)
• "How do I export my data?" (appears in 3% of tickets)

You should have 3-5 variants of each high-frequency question type, covering different phrasing:

• "How do I reset my password?"
• "How can I reset my password?"
• "Password reset procedure?"
• "I forgot my password, what do I do?"

Tier 2: Edge cases and boundary conditions

• "What if I have a negative balance?" (rare but financially sensitive)
• "Can I have multiple accounts?" (could cause confusion)
• "What if the export fails?" (error condition)

You should have 1-2 examples of each edge case category. Think about:

• Rare but important (security, compliance, legal).
• Error conditions (what if the data is corrupted?).
• Boundary conditions (empty account, very large dataset).
• Regional differences (policies vary by country).

Tier 3: Unanswerable questions

• "Will you integrate with X in the future?" (Feature request, not in docs.)
• "Why did you remove feature Y?" (History, not in current docs.)
• "How do I contact the CEO?" (Out of scope.)

Your RAG system should recognize and refuse these gracefully. Include 2-3 examples so you can score "correctly refused" vs "answered incorrectly".

Coverage target:

Total golden set: 100 questions

Tier 1 (high-frequency): 60 questions
  - Password reset: 5
  - Pricing: 8
  - Export/Import: 5
  - Account setup: 5
  - Billing: 5
  - Security/Privacy: 5
  - ...other high-frequency: 22

Tier 2 (edge cases): 30 questions
  - Boundary conditions: 8
  - Error recovery: 8
  - Regional differences: 8
  - Rare but important: 6

Tier 3 (unanswerable): 10 questions
  - Feature requests: 3
  - Out-of-scope: 4
  - Trick questions / adversarial: 3

This is a reasonable distribution for a small SaaS product. For a larger corpus, you might need 200-500 cases total, but the principle is the same.

Labeling ground truth

For each question, label:

1. Expected sources: Which document(s) should the system retrieve?
2. Acceptable answers: What are correct responses?
3. Metadata: Difficulty, category, why it matters.

Example format (JSON):

{
  "test_case_id": "pwd_reset_001",
  "query": "How do I reset my password?",
  "query_variants": [
    "How can I reset my password?",
    "Password reset procedure",
    "I forgot my password"
  ],
  "expected_sources": [
    {
      "source_id": "docs/account-management.md#password-reset",
      "document_name": "Account Management",
      "section": "Password Reset",
      "url": "https://docs.example.com/account-management#password-reset"
    }
  ],
  "acceptable_answers": [
    "Go to the login page, click 'Forgot password', and follow the email link.",
    "Use the 'Forgot password' option on the login page. Check your email for a reset link.",
    "Password reset is available via the login page. Click 'Forgot password'."
  ],
  "unacceptable_answers": [
    "Call customer support at 1-800-EXAMPLE.",
    "Use the 'Change password' option in settings (this is for logged-in users, not reset).",
    "We don't support password resets."
  ],
  "metadata": {
    "difficulty": "easy",
    "category": "account-management",
    "frequency_estimate": "5% of support tickets",
    "why_it_matters": "Very common question; if the RAG system fails here, users get frustrated quickly.",
    "added_date": "2026-07-01"
  },
  "notes": "Ensure the answer mentions the email reset link, not a direct link (for security)."
}

Example format (YAML, more concise):

- id: pricing_001
  query: What is the price of the Pro plan?
  expected_sources:
    - pricing.md#pro-tier
  acceptable_answers:
    - $99 per month
    - Pro plan costs $99/month
    - The Pro plan is $99 per month, billed annually at $1,188
  unacceptable_answers:
    - $99 per year
    - It depends on your usage
  difficulty: easy
  category: billing
  notes: Ensure the answer specifies monthly cost clearly

- id: export_edge_001
  query: "Can I export more than 1 million records?"
  expected_sources:
    - export-guide.md#limitations
  acceptable_answers:
    - "No, the export limit is 500k records per file"
    - "Maximum export is 500k records; use multiple exports for larger datasets"
  unacceptable_answers:
    - "Yes, you can export any amount"
    - "There is no limit"
  difficulty: hard
  category: edge-case
  notes: This is a boundary condition; test data includes accounts with >1M records

Who labels?

• Core team members (product, eng, support): They know what the right answer is.
• Subject matter experts: For domain-specific questions (legal, medical, financial).
• Avoid: Asking people unfamiliar with the product to label. They'll make mistakes.

Effort: ~5-10 minutes per test case to label ground truth thoroughly. For 100 cases, plan 8-16 hours. A team of 3 can do it in 1-2 days.

Versioning the golden set

Treat your golden set like production code: version it, track changes, and run tests against every version.

In Git:

content/evaluation/golden_set.json (or YAML)

Example commit history:

commit abc123: "Add 20 test cases from June support logs"
  + password reset variations (5 cases)
  + export/import edge cases (8 cases)
  + regional pricing differences (7 cases)

commit def456: "Update ground truth for password reset after docs change"
  ~ pwd_reset_001: Updated expected source from docs/faq.md to docs/account-management.md

commit ghi789: "Add 10 unanswerable adversarial cases"
  + trick question: "Are you sentient?" (should refuse)
  + out of scope: "Will you hire me?" (should refuse)
  ...

In a metrics dashboard:

Track performance over time:

Golden Set v1.0 (50 cases, frozen 2026-06-01):
  Baseline RAG system: 82% retrieval recall, 76% answer accuracy

Golden Set v1.1 (70 cases, frozen 2026-07-01):
  Same system: 80% retrieval recall, 74% answer accuracy
  (Score went down because we added harder edge cases)

Golden Set v1.2 (100 cases, frozen 2026-08-01):
  Same system: 79% retrieval recall, 72% answer accuracy
  (Larger, more diverse set)

After improvement A (better chunking):
  Golden Set v1.2: 85% retrieval recall, 78% answer accuracy

After improvement B (better prompt):
  Golden Set v1.2: 85% retrieval recall, 81% answer accuracy

This makes it clear that improvement B was worth doing, and improvement A was necessary but not sufficient.

Re-running the golden set on every change

Set up automated testing:

# evaluate.py (run in CI/CD)

import json
from pathlib import Path
from your_rag_system import retrieve_and_answer

def load_golden_set(filepath: str) -> list[dict]:
    """Load the golden set from JSON."""
    with open(filepath) as f:
        return json.load(f)

def evaluate_on_golden_set(system, test_cases: list[dict]) -> dict:
    """Run the system on each test case, score results."""
    results = {
        "total": len(test_cases),
        "passed": 0,
        "failed": 0,
        "by_category": {},
        "details": []
    }

    for case in test_cases:
        query = case["query"]
        expected_sources = case["expected_sources"]
        acceptable_answers = case["acceptable_answers"]

        # Get the system's answer
        answer, sources = system.answer(query)

        # Score retrieval: did we retrieve the expected sources?
        retrieved_ids = [s["source_id"] for s in sources]
        expected_ids = [s["source_id"] for s in expected_sources]
        retrieval_correct = any(eid in retrieved_ids for eid in expected_ids)

        # Score answer: is it in the acceptable list?
        answer_correct = any(
            acceptable in answer.lower()
            for acceptable in [a.lower() for a in acceptable_answers]
        )

        # Overall: both must be correct
        case_passed = retrieval_correct and answer_correct

        # Track results
        category = case["metadata"]["category"]
        if category not in results["by_category"]:
            results["by_category"][category] = {"passed": 0, "total": 0}

        results["by_category"][category]["total"] += 1
        if case_passed:
            results["passed"] += 1
            results["by_category"][category]["passed"] += 1
        else:
            results["failed"] += 1

        results["details"].append({
            "case_id": case["test_case_id"],
            "query": query,
            "passed": case_passed,
            "retrieval_correct": retrieval_correct,
            "answer_correct": answer_correct,
            "system_answer": answer
        })

    return results

if __name__ == "__main__":
    # Load golden set
    test_cases = load_golden_set("content/evaluation/golden_set.json")

    # Evaluate current system
    results = evaluate_on_golden_set(system, test_cases)

    # Print summary
    accuracy = results["passed"] / results["total"]
    print(f"Golden Set Evaluation Results")
    print(f"Total: {results['total']}")
    print(f"Passed: {results['passed']}")
    print(f"Failed: {results['failed']}")
    print(f"Accuracy: {accuracy:.1%}")

    # Print by category
    for category, scores in results["by_category"].items():
        cat_accuracy = scores["passed"] / scores["total"]
        print(f"  {category}: {cat_accuracy:.1%} ({scores['passed']}/{scores['total']})")

    # Print failures for inspection
    failures = [d for d in results["details"] if not d["passed"]]
    if failures:
        print("\nFailed cases:")
        for f in failures[:10]:  # Show first 10
            print(f"  {f['case_id']}: {f['query']}")
            print(f"    Expected retrieval: {f['retrieval_correct']}")
            print(f"    Expected answer: {f['answer_correct']}")

    # Exit with non-zero if accuracy drops below threshold
    if accuracy < 0.80:
        exit(1)

In CI/CD (e.g., GitHub Actions):

name: RAG Golden Set Evaluation

on: [pull_request]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run golden set evaluation
        run: python evaluate.py
        env:
          GOLDEN_SET_PATH: content/evaluation/golden_set.json

      - name: Comment PR with results
        if: always()
        uses: actions/github-script@v6
        with:
          script: |
            // Parse evaluation results and post as PR comment
            const fs = require('fs');
            const results = JSON.parse(fs.readFileSync('evaluation_results.json'));
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `Golden Set Evaluation: ${results.passed}/${results.total} (${(results.passed/results.total*100).toFixed(1)}%)`
            });

Now, every PR that touches the RAG system automatically runs the golden set and reports results.

Maintaining the golden set over time

Every month:

• Mine new support/search logs.
• Add 5-10 new test cases from real queries you didn't have coverage for.
• Re-evaluate the system on the full set.

Every quarter:

• Review failed cases. Are they failures in your system, or mislabeled ground truth?
• Update ground truth if docs changed (e.g., "Password reset changed in docs on July 15").
• Remove duplicate or near-duplicate cases.

When shipping major changes:

• Run the golden set before the change.
• Run it after the change.
• Compare. If scores drop, investigate before shipping.

Practice: Build a small golden set

Scenario: You're building RAG for a SaaS support system with 5 core features.

Step 1: Mine 50 real questions

From the past month's support logs, extract questions:

Feature: Account Setup
  Q1: "How do I create an account?"
  Q2: "Can I use the same email for multiple accounts?"
  Q3: "What information do I need to set up?"

Feature: Billing
  Q4: "What are your prices?"
  Q5: "Do you offer annual discounts?"
  Q6: "Can I change my plan later?"
  ...

Step 2: Curate to 30 (70% tier 1, 20% tier 2, 10% tier 3)

Tier 1 (21):
  Account setup: 3
  Billing: 3
  Exporting: 3
  ...

Tier 2 (6):
  Regional pricing differences: 2
  Large account edge cases: 2
  Error recovery: 2

Tier 3 (3):
  Out of scope: 2
  Adversarial / trick: 1

Step 3: Label each with ground truth

For each, identify:

• Expected doc section
• Acceptable answers (2-3 variants)
• Unacceptable answers (2-3 common wrong answers)

Step 4: Version and baseline

content/evaluation/golden_set_v1.0.json

Run your current RAG system on all 30, record baseline scores:

• Retrieval recall: 85%
• Answer correctness: 78%

Step 5: CI/CD

Set up evaluate.py to run on every PR. Now you have a objective measure: "Does this change help or hurt?"

Common mistake

Building a golden set once and never updating it. Your corpus grows, user behavior changes, and edge cases emerge. A golden set from 6 months ago is stale. It doesn't reflect production traffic patterns anymore.

Also: Being too strict with labeling. "The answer must be word-for-word from the document." But users don't ask word-for-word questions, and RAG systems paraphrase naturally. Define acceptable answers generously: "The answer is correct if it conveys the key information, even if phrased differently."

And: Testing only the happy path. Include unanswerable questions, edge cases, and adversarial inputs. If 5% of production traffic is "trick questions," your golden set should include similar cases.