Evaluate Retrieval and Answers Separately

The problem with end-to-end scoring

Many teams score RAG systems with a single metric: "Is the final answer correct?" This seems intuitive—if the answer is right, everything worked. If it's wrong, something failed.

But this single score obscures where the failure actually occurred. Consider this scenario:

Test case: "Who is the VP of Engineering?"

Expected answer: "Alice Chen"

System output: "Based on our documents, the VP of Engineering is Bob Smith."

Score: INCORRECT (Alice, not Bob).

But where did it go wrong?

• Option A: Retrieval failed. We retrieved an outdated org chart from 2024, which listed Bob. The correct 2025 org chart exists in the corpus but wasn't retrieved.
• Option B: Retrieval was perfect. We retrieved the correct 2025 org chart (which says Alice). But the model misread it or preferred the 2024 version it also retrieved.
• Option C: Both. Retrieval returned a mix of old and new, and the model had no way to know which was current.

These are three completely different failure modes. The fix for A is different (improve recency filtering), different from B (better prompt), different from C (both).

With a single end-to-end score, you don't know which one is true. You try all three fixes, hoping something works. With separate retrieval and generation scores, the answer is immediate.

Scoring the retrieval layer independently

Retrieval scoring is straightforward: Did we retrieve the right documents?

This is independent of what the model later says. If the correct answer is in the retrieved chunks, retrieval succeeded, even if the model botches the answer.

Key metrics

Recall @ K:

"Of all the relevant documents in the corpus, did we retrieve at least one of them in our top K results?"

def recall_at_k(relevant_docs: set[str], retrieved_docs: list[str], k: int) -> float:
    """
    relevant_docs: Set of document IDs that correctly answer the query
    retrieved_docs: List of (document_id, score) tuples returned by retrieval, ordered by score
    k: How many top results to consider
    """
    top_k_ids = set(doc_id for doc_id, _ in retrieved_docs[:k])
    matches = relevant_docs & top_k_ids
    if not relevant_docs:
        return 1.0  # No relevant docs = vacuously true
    return len(matches) / len(relevant_docs)

# Example
relevant = {"docs/org_chart_2025.pdf", "docs/leadership.md"}
retrieved = [
    ("docs/org_chart_2024.pdf", 0.89),
    ("docs/leadership.md", 0.87),
    ("docs/team_page.html", 0.72),
    ("docs/about_us.md", 0.68),
    ("docs/org_chart_2025.pdf", 0.65),
]

recall_1 = recall_at_k(relevant, retrieved, k=1)  # 0.0 (missed both relevant docs)
recall_5 = recall_at_k(relevant, retrieved, k=5)  # 1.0 (retrieved both by position 5)

For a production system, recall_at_5 should be > 85% at minimum. If it's 60%, your retrieval is the bottleneck.

Precision @ K:

"Of the top K documents we retrieved, how many were actually relevant?"

def precision_at_k(relevant_docs: set[str], retrieved_docs: list[str], k: int) -> float:
    """
    Same parameters as recall_at_k.
    """
    top_k_ids = set(doc_id for doc_id, _ in retrieved_docs[:k])
    matches = relevant_docs & top_k_ids
    return len(matches) / k

# Example
precision_1 = precision_at_k(relevant, retrieved, k=1)  # 0.0 (first result is irrelevant)
precision_5 = precision_at_k(relevant, retrieved, k=5)  # 0.4 (2 out of 5 are relevant)

Precision is crucial for user experience: if 50% of your top 5 results are junk, users waste time reading irrelevant documents.

Mean Reciprocal Rank (MRR):

"How early in the ranked list is the first relevant document?"

def mrr(relevant_docs: set[str], retrieved_docs: list[str]) -> float:
    """
    Rewards retrieval systems that place relevant docs near the top.
    """
    for rank, (doc_id, _) in enumerate(retrieved_docs, 1):
        if doc_id in relevant_docs:
            return 1.0 / rank  # First match at rank 1 = MRR 1.0, rank 5 = MRR 0.2
    return 0.0  # No relevant doc found

# Example
mrr = mrr(relevant, retrieved)  # 0.5 (first relevant doc at rank 2: 1/2)

MRR of 0.9+ means relevant docs are showing up in the top 2-3. Below 0.7, users might not find what they need.

Normalized Discounted Cumulative Gain (nDCG):

Like MRR, but with a twist: relevance is a score, not binary. A partially relevant doc might score 0.5, a highly relevant doc 1.0.

def ndcg(ideal_ranking: list[float], actual_ranking: list[float]) -> float:
    """
    ideal_ranking: Perfect ranking scores [1.0, 1.0, 0.5, 0.0, ...]
    actual_ranking: Our system's ranking scores [1.0, 0.5, 0.0, 1.0, ...]

    DCG rewards good docs early, penalizes good docs late.
    nDCG normalizes DCG by the ideal (best possible) ordering.
    """
    def dcg(scores):
        return sum(score / (i + 1) for i, score in enumerate(scores))

    actual_dcg = dcg(actual_ranking[:10])  # Consider top 10
    ideal_dcg = dcg(sorted(ideal_ranking[:10], reverse=True))

    if ideal_dcg == 0:
        return 1.0
    return actual_dcg / ideal_dcg

# Example
ideal = [1.0, 1.0, 0.5, 0.0, 0.0]  # 2 perfect, 1 partial, 2 irrelevant
actual = [1.0, 0.5, 0.0, 1.0, 0.0]  # 1 perfect, 1 partial, 1 irrelevant, 1 perfect late
nDCG = ndcg(ideal, actual)  # ~0.92 (good but not perfect)

Bottom line for retrieval:

• Recall @ 5 ≥ 85% (are we finding the right documents at all?)
• Precision @ 5 ≥ 60% (are we returning mostly relevant results?)
• MRR ≥ 0.7 (are relevant docs near the top?)
• nDCG @ 10 ≥ 0.8 (are we ranking well?)

If any of these dip below threshold, the retrieval layer is your problem. Tune chunking, improve embedding model, or switch to hybrid search.

Scoring the generation layer independently

Generation scoring is: Given the retrieved documents, is the answer good?

This assumes the retrieved documents are fixed and correct. You're evaluating only the model's ability to read and synthesize.

Key metrics

Faithfulness (also called "groundedness"):

"Is every claim in the answer supported by the retrieved documents?"

This is the most important metric. An answer that sounds good but contradicts the source is worse than an answer that's incomplete.

def score_faithfulness(answer: str, retrieved_chunks: list[str], llm) -> dict:
    """
    Use an LLM to score each claim in the answer.
    """
    prompt = f"""
You are a fact-checker. For each claim in the answer, determine if it is:
- SUPPORTED: The retrieved documents clearly state this
- UNSUPPORTED: The documents don't mention it (but don't contradict it)
- CONTRADICTED: The documents explicitly deny this

Answer: "{answer}"

Retrieved documents:
{chr(10).join(f"- {chunk[:200]}..." for chunk in retrieved_chunks)}

Score each claim:
1. [claim 1]: SUPPORTED/UNSUPPORTED/CONTRADICTED
2. [claim 2]: ...

Then give an overall faithfulness score (0-1).
"""

    result = llm.generate(prompt)
    # Parse result to extract per-claim scores and overall score
    return {
        "per_claim_scores": {...},
        "overall_faithfulness": 0.85
    }

# Example
answer = "The VP of Engineering is Alice Chen. She joined in 2020."
chunks = ["Alice Chen is the VP of Engineering...", "She started in August 2020..."]
faithfulness = score_faithfulness(answer, chunks, llm)
# Result: {"overall_faithfulness": 1.0} (both claims supported)

Target: ≥ 0.90 (at least 90% of claims must be supported or unsupported; contradictions are rare).

Relevance:

"Does the answer actually answer the question, or is it off-topic?"

def score_relevance(question: str, answer: str, llm) -> float:
    """
    Does the answer address the question?
    """
    prompt = f"""
Question: "{question}"
Answer: "{answer}"

Rate the relevance of the answer to the question on a scale of 0-1:
- 0: Completely off-topic
- 0.3: Tangentially related
- 0.6: Partially answers the question
- 0.9: Mostly answers the question
- 1.0: Fully and directly answers the question

Respond with just the score (0-1).
"""

    score_str = llm.generate(prompt).strip()
    return float(score_str)

# Example
question = "What is the return policy?"
answer = "Our company was founded in 2015 and has 500 employees."
relevance = score_relevance(question, answer, llm)  # 0.0 (completely off-topic)

answer = "We accept returns within 30 days of purchase for a full refund."
relevance = score_relevance(question, answer, llm)  # 1.0 (directly answers)

Target: ≥ 0.85 (answers should address the question, not go on tangents).

Completeness:

"Does the answer cover all parts of the question?"

def score_completeness(question: str, answer: str, llm) -> float:
    """
    For multi-part questions, does the answer cover all parts?
    """
    prompt = f"""
Question: "{question}"
Answer: "{answer}"

Does the answer address all parts of the question?
For example, if the question has 3 sub-parts (a), (b), (c),
does the answer address all three?

Rate completeness 0-1:
- 0: Addresses none of the sub-parts
- 0.5: Addresses some but not all
- 1.0: Addresses all parts

Respond with just the score (0-1).
"""

    score_str = llm.generate(prompt).strip()
    return float(score_str)

# Example
question = "What is the return policy, shipping time, and warranty?"
answer = "Returns are accepted within 30 days for a full refund."
completeness = score_completeness(question, answer, llm)  # 0.33 (only 1 of 3 parts)

Target: ≥ 0.8 (most questions should be fully addressed).

Citation accuracy:

"Does the answer cite sources, and do those sources actually appear in the retrieved set?"

def score_citation_accuracy(answer: str, retrieved_chunks: list[str], llm) -> float:
    """
    Count citations that are verifiable in the retrieved chunks.
    """
    prompt = f"""
Answer: "{answer}"

Retrieved documents:
{chr(10).join(f"[{i}] {chunk[:300]}..." for i, chunk in enumerate(retrieved_chunks))}

For each claim in the answer that should be cited, verify:
1. Is there a citation (e.g., [1], [2])?
2. Does the cited chunk actually support the claim?

Score: (# of correctly cited claims) / (# of claims that should be cited)
"""

    result = llm.generate(prompt)
    # Parse to get a 0-1 score
    return 0.92

# Example
answer = "The refund window is 30 days [1]. This applies globally [2]."
chunks = ["[Refund policy doc] The refund window is 30 days...", "[Regional policy] US and EU have 30 days; other regions have 14 days..."]
citation_accuracy = score_citation_accuracy(answer, chunks, llm)  # 0.5 (first citation correct, second is wrong)

Target: ≥ 0.95 (citations should be trustworthy).

Bottom line for generation:

• Faithfulness ≥ 0.90 (avoid contradictions)
• Relevance ≥ 0.85 (stay on topic)
• Completeness ≥ 0.80 (address all parts)
• Citation accuracy ≥ 0.95 (citations are trustworthy)

If generation metrics are low, improve prompting, add constraints ("Only cite retrieved chunks"), or use a better model.

Controlled comparisons: isolating which layer changed

Here's where separate scoring becomes powerful. You can run controlled experiments.

Experiment 1: Same retrieval, different prompts

Setup:

• Retrieve once using your baseline retrieval system.
• Fix the retrieved chunks.
• Compare two prompts (old vs new).

def controlled_comparison_prompt(
    golden_set: list[dict],
    retrieval_system,
    prompt_old: str,
    prompt_new: str,
    llm
) -> dict:
    """
    Compare two prompts with identical retrieval results.
    """
    results = {"old": {}, "new": {}}

    for test_case in golden_set:
        query = test_case["query"]

        # Step 1: Retrieve (identical for both prompts)
        chunks = retrieval_system.search(query, top_k=5)

        # Step 2: Generate with old prompt
        answer_old = llm.generate(prompt_old.format(
            context=chunks,
            question=query
        ))

        # Step 3: Generate with new prompt
        answer_new = llm.generate(prompt_new.format(
            context=chunks,
            question=query
        ))

        # Step 4: Score both
        faithfulness_old = score_faithfulness(answer_old, chunks, llm)
        faithfulness_new = score_faithfulness(answer_new, chunks, llm)

        relevance_old = score_relevance(query, answer_old, llm)
        relevance_new = score_relevance(query, answer_new, llm)

        # Accumulate results
        results["old"][test_case["id"]] = {
            "faithfulness": faithfulness_old,
            "relevance": relevance_old
        }
        results["new"][test_case["id"]] = {
            "faithfulness": faithfulness_new,
            "relevance": relevance_new
        }

    # Aggregate
    avg_old_faith = sum(r["faithfulness"] for r in results["old"].values()) / len(results["old"])
    avg_new_faith = sum(r["faithfulness"] for r in results["new"].values()) / len(results["new"])
    print(f"Faithfulness: Old {avg_old_faith:.2f} -> New {avg_new_faith:.2f}")

    return results

Conclusion: If new prompt scores 0.92 and old scores 0.89, the new prompt is better at faithfulness. The improvement is entirely due to the prompt change, because retrieval was identical.

Experiment 2: Same prompt, different retrieval

Setup:

• Retrieve with two different configs (old vs new chunking, new embedding model, etc.).
• Fix the prompt.
• Compare retrieval quality metrics.

def controlled_comparison_retrieval(
    golden_set: list[dict],
    retrieval_system_old,
    retrieval_system_new,
    prompt: str,
    llm
) -> dict:
    """
    Compare two retrieval systems with an identical prompt/generation step.
    """
    results = {"old": {}, "new": {}}

    for test_case in golden_set:
        query = test_case["query"]
        expected_sources = test_case["expected_sources"]

        # Retrieve with old system
        chunks_old = retrieval_system_old.search(query, top_k=5)
        recall_old = recall_at_k(expected_sources, chunks_old, k=5)
        precision_old = precision_at_k(expected_sources, chunks_old, k=5)

        # Retrieve with new system
        chunks_new = retrieval_system_new.search(query, top_k=5)
        recall_new = recall_at_k(expected_sources, chunks_new, k=5)
        precision_new = precision_at_k(expected_sources, chunks_new, k=5)

        # Generate with prompt (identical for both)
        answer_old = llm.generate(prompt.format(context=chunks_old, question=query))
        answer_new = llm.generate(prompt.format(context=chunks_new, question=query))

        # Score generation
        faithfulness_old = score_faithfulness(answer_old, chunks_old, llm)
        faithfulness_new = score_faithfulness(answer_new, chunks_new, llm)

        results["old"][test_case["id"]] = {
            "recall": recall_old,
            "precision": precision_old,
            "faithfulness": faithfulness_old
        }
        results["new"][test_case["id"]] = {
            "recall": recall_new,
            "precision": precision_new,
            "faithfulness": faithfulness_new
        }

    # Aggregate and compare
    avg_old_recall = sum(r["recall"] for r in results["old"].values()) / len(results["old"])
    avg_new_recall = sum(r["recall"] for r in results["new"].values()) / len(results["new"])

    print(f"Retrieval Recall: Old {avg_old_recall:.2f} -> New {avg_new_recall:.2f}")
    print(f"Impact on faithfulness: Old {avg_old_faith:.2f} -> New {avg_new_faith:.2f}")

    return results

Conclusion: If new retrieval achieves 87% recall vs old's 82%, the improvement cascades to generation. Better chunks → better answers.

Combining automated metrics with human calibration

Automated metrics are fast but imperfect. Humans are slow but accurate. Use both.

Workflow:

1. Run automated metrics on a large sample (100+ test cases). This is cheap.
2. Sample a subset (20% of cases) for human review.
3. Measure disagreement between automated and human judgments.
4. Refine the automated metric if disagreement is high.
5. Scale: Once calibrated, use automated metrics confidently.

def calibrate_faithfulness_metric(
    golden_set: list[dict],
    retrieval_system,
    llm,
    human_judges: int = 3,
    sample_size: int = 20
):
    """
    Calibrate the automated faithfulness scorer against human judgment.
    """

    # Step 1: Run automated faithfulness on all cases
    automated_scores = {}
    for case in golden_set:
        chunks = retrieval_system.search(case["query"], top_k=5)
        answer = llm.generate(case["prompt"].format(
            context=chunks,
            question=case["query"]
        ))
        automated_score = score_faithfulness(answer, chunks, llm)
        automated_scores[case["id"]] = automated_score

    # Step 2: Sample 20% for human review
    sample_ids = random.sample(list(automated_scores.keys()), sample_size)
    human_scores = {}

    for case_id in sample_ids:
        case = next(c for c in golden_set if c["id"] == case_id)
        chunks = retrieval_system.search(case["query"], top_k=5)
        answer = llm.generate(case["prompt"].format(
            context=chunks,
            question=case["query"]
        ))

        # Each case is judged by 3 humans
        scores = []
        for judge in range(human_judges):
            score = human_judge(
                question=case["query"],
                answer=answer,
                chunks=chunks
            )
            scores.append(score)

        # Average human scores
        human_scores[case_id] = sum(scores) / len(scores)

    # Step 3: Measure disagreement
    disagreements = []
    for case_id in sample_ids:
        auto = automated_scores[case_id]
        human = human_scores[case_id]
        disagreement = abs(auto - human)
        disagreements.append(disagreement)

        if disagreement > 0.2:
            print(f"Case {case_id}: Automated {auto:.2f}, Human {human:.2f} -- DISAGREEMENT")

    avg_disagreement = sum(disagreements) / len(disagreements)
    print(f"Average disagreement: {avg_disagreement:.3f}")

    # Step 4: Decide if metric is trustworthy
    if avg_disagreement < 0.10:
        print("✓ Metric is well-calibrated. Use automated scoring confidently.")
    elif avg_disagreement < 0.20:
        print("~ Metric is moderately calibrated. Use with caution; spot-check regularly.")
    else:
        print("✗ Metric is poorly calibrated. Refine the metric before using at scale.")

    return {
        "automated": automated_scores,
        "human": human_scores,
        "avg_disagreement": avg_disagreement,
        "calibration_verdict": "well-calibrated" if avg_disagreement < 0.10 else "needs work"
    }

A complete worked example

Scenario: Your RAG system scores 75% on answer correctness. You want to improve it. But is the problem retrieval or generation?

Setup:

• Golden set: 100 test cases
• Baseline: Retrieval recall 82%, generation faithfulness 88%, end-to-end correctness 75%

Analysis:

If retrieval recall is 82%, then ~18% of questions have no relevant chunk in the top 5. Those questions cannot be answered correctly, no matter how good your prompt is. This is the ceiling for end-to-end performance: max 82%.

Your actual end-to-end score is 75%, which means:

• Of the 82% where retrieval succeeded, only 75% / 82% ≈ 91% result in a correct answer.
• So generation is ~91% effective.

Diagnosis:

1. Retrieval is the bottleneck. 18% of failures are due to bad retrieval. Improving retrieval from 82% recall to 90% recall would improve end-to-end by ~8%.
2. Generation is reasonable. 91% of cases where retrieval succeeds result in a correct answer.

Action: Focus on retrieval. Try:

• Better chunking (keep full context, don't split mid-sentence)
• Hybrid search (add keyword search alongside embeddings)
• Better embedding model (a larger model might distinguish more subtly related documents)

Validation:

• Implement new retrieval config
• Measure retrieval recall alone (don't change prompt): Does it improve to 88%?
• Measure end-to-end: Does it improve to 82%?

If retrieval improved but end-to-end didn't, generation is now the bottleneck. Pivot to improving the prompt.

Building confidence in your metrics over time

The first time you measure faithfulness, you'll find disagreement between your automated scorer and human judges. This is normal. The goal isn't perfect agreement (impossible), but enough agreement to be actionable.

A practical approach: every 2 weeks, sample 10-20 test cases at random, have 2-3 humans score them independently, and compare to your automated metrics. Track disagreement over time. If you started at 40% disagreement and are now at 15%, your metrics are becoming more trustworthy. Use those slices to understand which edge cases your metrics struggle with (e.g., "metrics are good on factual questions, weak on subjective judgment calls") and either refine the metric or accept the limitation and handle those cases with human review.

This iterative refinement builds an evaluation system that you can trust and explain to stakeholders.

Common mistake

Conflating retrieval quality with generation quality. A system might have 95% retrieval recall but 60% end-to-end correctness because the prompt is terrible at synthesizing. Or vice versa: perfect retrieval with a bad prompt still fails.

Also: Optimizing for the wrong metric. If you increase precision at the cost of recall, you might make the top 5 results more relevant but delete the actually-correct answer entirely. Measure the interaction.

Finally: Not replicating human judgment. An automated metric that disagrees with humans 30% of the time is wrong, no matter how fast it is. Always calibrate against human labels before deploying at scale.