Retrieval Metrics: Recall, Precision, MRR, and nDCG

Understanding retrieval metrics

When you deploy a RAG system, you need to measure one thing: Did retrieval give the model the information it needed to answer the user's question? But that simple question has many answers, each valuable in different contexts.

Four metrics dominate RAG evaluation, each answering a slightly different version of that question:

• Recall@k: "Of all the relevant chunks that exist, did we find at least one in the top k?"
• Precision@k: "Of the k chunks we returned, what fraction were actually relevant?"
• MRR (Mean Reciprocal Rank): "On average, how far down the ranked list was the first relevant chunk?"
• nDCG (normalized Discounted Cumulative Gain): "How much did we penalize retrieving relevant chunks late in the list?"

Each metric has a different implicit assumption about what "good retrieval" means. Understanding these differences will help you pick the right metric for your system and know when your metric might be lying to you.

Recall@k: "Did we find it?"

Definition: The fraction of all relevant documents that appear somewhere in the top k results.

Recall@k = (# of relevant docs in top k) / (# of relevant docs total)

Example: Your golden set contains a query with 3 relevant chunks. You retrieve the top 10 results. If 2 of those 3 chunks appear in the top 10, your Recall@10 = 2/3 ≈ 0.67.

def compute_recall_at_k(retrieved_chunk_ids, relevant_chunk_ids, k):
    """
    Args:
        retrieved_chunk_ids: list of chunk IDs in rank order (top to bottom)
        relevant_chunk_ids: set of all chunk IDs that are correct for this query
        k: the cutoff (e.g., k=10 means top 10)

    Returns:
        float: Recall@k
    """
    if not relevant_chunk_ids:
        return 1.0  # edge case: no relevant chunks means perfect recall

    top_k = set(retrieved_chunk_ids[:k])
    hits = len(top_k & relevant_chunk_ids)  # intersection
    return hits / len(relevant_chunk_ids)

When Recall@k matters:

• You need to ensure the model always has the right chunks available, even if they're buried in a larger result set
• Your use case can tolerate some irrelevant chunks (the model is good at filtering them)
• Example: A customer-support chatbot where missing the right policy is worse than including 20 irrelevant policies

The recall trap: Recall alone doesn't tell you about noise. A system that returns all 1000 chunks would have Recall@1000 = 1.0, but Recall@5 might be 0.6. Always report Recall@k and specify k. Reporting just "Recall" is meaningless.

Recall also scales with the number of relevant documents. A query with 1 relevant chunk can't drop below Recall@k = 0 or above 1.0. A query with 50 relevant chunks is much harder to satisfy fully (you need many top-k slots), so comparing recall across queries with different ground-truth set sizes can be misleading.

Precision@k: "How much noise is acceptable?"

Definition: The fraction of the top k results that are actually relevant.

Precision@k = (# of relevant docs in top k) / k

Example: You retrieve top 10 results. 3 of them are relevant. Precision@10 = 3/10 = 0.30. (This means 70% of what you're showing the model is noise.)

def compute_precision_at_k(retrieved_chunk_ids, relevant_chunk_ids, k):
    """
    Args:
        retrieved_chunk_ids: list of chunk IDs in rank order
        relevant_chunk_ids: set of all chunk IDs that are correct
        k: the cutoff

    Returns:
        float: Precision@k
    """
    top_k = set(retrieved_chunk_ids[:k])
    hits = len(top_k & relevant_chunk_ids)
    return hits / k

When Precision@k matters:

• Your prompt is expensive: you want every retrieved chunk to "count" toward the answer
• The model struggles to filter noise: each irrelevant chunk wastes tokens and risks confusing the model
• Example: A medical diagnosis assistant where hallucinated symptoms are worse than incomplete information

The precision trap: High Precision@k can hide poor recall. A system that retrieves only 1 chunk, and that chunk is relevant, has Precision@1 = 1.0 (perfect!), but Recall@1 = 1/50 = 0.02 if there were 50 relevant chunks. You missed 98% of the relevant information but precision looks perfect.

MRR: "How confident was the first hit?"

Definition: The mean of 1 / (rank of first relevant result), averaged across many queries.

MRR = (1 / |Q|) × Σ [1 / (rank of first relevant doc in query i)]

Example: For a single query, if the first relevant chunk appears at rank 3, the reciprocal rank is 1/3 ≈ 0.33. If it's at rank 1, MRR contribution is 1/1 = 1.0. If no relevant chunk is found in the top k, the contribution is 0.

def compute_mrr(queries_results, k):
    """
    Args:
        queries_results: list of (retrieved_chunk_ids, relevant_chunk_ids) tuples
        k: the cutoff for search depth

    Returns:
        float: mean reciprocal rank across all queries
    """
    rr_sum = 0
    for retrieved_chunk_ids, relevant_chunk_ids in queries_results:
        for rank, chunk_id in enumerate(retrieved_chunk_ids[:k], start=1):
            if chunk_id in relevant_chunk_ids:
                rr_sum += 1 / rank
                break
        # if no relevant chunk found, contributes 0 to the sum

    return rr_sum / len(queries_results)

When MRR matters:

• The first result matters most; subsequent results add diminishing value
• Use case: "Did the top result solve the user's problem?" (e.g., a search engine where users click the first link)
• You want a single metric that rewards early hits

Why MRR is elegant but underused: MRR gives a single number (0 to 1) that naturally rewards putting the best result first. A system where all relevant results are at position 1 has MRR = 1.0. A system where relevant results are scattered (rank 5, 10, 15 on different queries) has a lower, more honest MRR.

But MRR only cares about one relevant result per query. If there are 50 relevant chunks and you retrieve 30 of them, MRR only "sees" which rank the first one came in at, and ignores the other 29. This makes MRR less useful for systems where multiple relevant chunks are genuinely needed.

nDCG: "Ranking quality matters"

Definition: Discounted Cumulative Gain, normalized by the ideal ordering.

This metric has two parts:

Cumulative Gain (CG): Sum the relevance scores of all chunks in the top k.

CG@k = Σ(i=1 to k) rel(i)

where rel(i) is 1 if chunk i is relevant, 0 otherwise.

Discounted Cumulative Gain (DCG): Apply a discount factor that decreases with rank.

DCG@k = Σ(i=1 to k) [rel(i) / log₂(i + 1)]

The log discount means position 1 has no discount (log₂(2) = 1), position 2 is divided by log₂(3) ≈ 1.58, position 10 is divided by log₂(11) ≈ 3.46, and so on. Relevant chunks retrieved later are worth less.

Normalized DCG (nDCG): Divide DCG by the best possible DCG (if chunks were ranked perfectly).

nDCG@k = DCG@k / Ideal DCG@k

where Ideal DCG is the DCG you'd get if you ranked all relevant chunks first, then all irrelevant ones.

Example:

Query: "What is the refund policy?"
Retrieved: [chunk_A (relevant), chunk_B (irrelevant), chunk_C (relevant), chunk_D (irrelevant), chunk_E (relevant)]

nDCG@5:
- DCG = 1/log₂(2) + 0/log₂(3) + 1/log₂(4) + 0/log₂(5) + 1/log₂(6)
       = 1/1 + 0 + 1/2 + 0 + 1/2.58
       = 1 + 0 + 0.5 + 0 + 0.39
       = 1.89

- Ideal DCG (best possible with 3 relevant chunks):
       = 1/log₂(2) + 1/log₂(3) + 1/log₂(4)
       = 1 + 0.63 + 0.5
       = 2.13

- nDCG@5 = 1.89 / 2.13 ≈ 0.89

import math

def compute_ndcg_at_k(retrieved_chunk_ids, relevant_chunk_ids, k):
    """
    Args:
        retrieved_chunk_ids: list of chunk IDs in rank order
        relevant_chunk_ids: set of relevant chunk IDs
        k: cutoff

    Returns:
        float: nDCG@k (0 to 1)
    """
    # Compute actual DCG
    dcg = 0
    for rank, chunk_id in enumerate(retrieved_chunk_ids[:k], start=1):
        if chunk_id in relevant_chunk_ids:
            # Discount by log of (rank + 1)
            dcg += 1 / math.log2(rank + 1)

    # Compute ideal DCG (all relevant docs ranked first)
    ideal_dcg = 0
    for rank in range(1, min(len(relevant_chunk_ids) + 1, k + 1)):
        ideal_dcg += 1 / math.log2(rank + 1)

    # Avoid division by zero
    if ideal_dcg == 0:
        return 1.0 if dcg == 0 else 0.0

    return dcg / ideal_dcg

When nDCG matters:

• You care about ranking quality: relevant chunks early are much better than relevant chunks late
• The model will read the top-k results in order; the top results dominate its reasoning
• Example: A document search where the first 3 results are read carefully but the rest are skimmed

Why nDCG is the gold standard: nDCG normalizes to [0, 1], making it comparable across queries with different numbers of relevant chunks. It rewards both having relevant results and ranking them high. A system with Recall@10 = 1.0 but all relevant chunks at rank 10 has low nDCG; a system with Recall@10 = 0.9 but all hits at rank 1-2 has high nDCG. This aligns with real-world user behavior: position matters.

Choosing k: the hidden cost

All four metrics depend on k. Smaller k means lower cost (fewer chunks to embed, fewer tokens to the model) but higher risk of missing information. Larger k means completeness but waste.

# Real-world scenario: query with 5 truly relevant chunks out of 10,000 total

retrieved = [
    # rank 1-2: relevant
    "chunk_42", "chunk_88",
    # rank 3-8: irrelevant
    "chunk_101", "chunk_203", "chunk_404", "chunk_505",
    # rank 9-10: relevant
    "chunk_19", "chunk_77",
    # rank 11+: irrelevant
    # ...
]
relevant = {"chunk_42", "chunk_88", "chunk_19", "chunk_77", "chunk_156"}  # 5 relevant, one is at rank > 20

# At different k values:
print(f"Recall@5 = {2/5} = 0.40")      # only top 2 relevant chunks
print(f"Recall@10 = {4/5} = 0.80")     # missed chunk_156 which is at rank 23
print(f"Recall@20 = {4/5} = 0.80")     # still missed chunk_156
print(f"Recall@25 = {5/5} = 1.00")     # all relevant chunks now found

print(f"Precision@5 = {2/5} = 0.40")   # 60% noise
print(f"Precision@10 = {4/10} = 0.40") # 60% noise still
print(f"Precision@20 = {4/20} = 0.20") # 80% noise

print(f"MRR = {1/1} = 1.0")            # first relevant at rank 1

print(f"nDCG@5 = 0.63")                # two relevant in top 5, both early
print(f"nDCG@10 = 0.75")               # four relevant in top 10, some later
print(f"nDCG@25 = 0.78")               # all five relevant, but one late

Choosing k in practice:

• k=5: Fast, cheap (5-10 chunks in prompt). Good for simple queries where 1-2 relevant chunks suffice. Risk: recall is incomplete.
• k=10: Standard in most systems. Balances cost and completeness. Good default.
• k=20+: For complex, multi-faceted questions where multiple angles need to be covered. Higher token cost and more noise.

The token budget constraint: If your prompt template is 500 tokens (system message + user query), and each chunk is 300 tokens, you can fit k ≈ 4 chunks before hitting token limits. Even if nDCG@10 is perfect, you can only use 4 results. In this case, report both Recall@10 and Recall@4 (the actual bottleneck). This reveals the mismatch between what retrieval can do and what the prompt architecture allows.

A complete worked example: evaluating a customer-support RAG

Your customer-support chatbot retrieves policy chunks to answer "How do I cancel my subscription?"

Golden set for this query:

• Relevant chunks: "Cancellation policy" (ID 42), "Account settings menu" (ID 88), "Billing FAQ — cancellation" (ID 19)
• Total relevant: 3

System's retrieval (top 15 results):

Rank  Chunk ID   Title                              Relevant?
1     42         Cancellation policy                YES
2     99         Refund processing times            NO
3     88         Account settings menu              YES
4     105        How to update payment method       NO
5     203        Subscription tiers                 NO
6     19         Billing FAQ — cancellation         YES
7     144        Contact support                    NO
... (ranks 8-15 irrelevant)

Computing metrics:

retrieved_ids = [42, 99, 88, 105, 203, 19, 144, ...]
relevant_ids = {42, 88, 19}

# Recall@5
recall_at_5 = 2 / 3  # chunks 42, 88 found; missing 19
# = 0.67

# Recall@10
recall_at_10 = 3 / 3  # all three found by rank 6
# = 1.0

# Precision@5
precision_at_5 = 2 / 5  # ranks 1 and 3 are relevant; 3 are not
# = 0.40

# Precision@10
precision_at_10 = 3 / 10
# = 0.30

# MRR
mrr = 1 / 1  # first relevant is at rank 1
# = 1.0

# nDCG@10
dcg = 1/log2(2) + 1/log2(4) + 1/log2(6)
    = 1 + 0.5 + 0.387
    = 1.887
ideal_dcg = 1/log2(2) + 1/log2(3) + 1/log2(4)
          = 1 + 0.631 + 0.5
          = 2.131
ndcg_at_10 = 1.887 / 2.131
           = 0.885

Interpretation:

• Recall@10 = 1.0: Good, all relevant chunks found
• Precision@10 = 0.30: Fair, 70% of retrieved chunks are noise, but acceptable if the model filters well
• MRR = 1.0: Excellent, first result was right
• nDCG@10 = 0.89: Very good, most relevant chunks ranked high, with one gap at rank 6

Decision: This is a strong retrieval result. The query can be answered from the top 5-6 chunks, with good ranking of the most important ones.

Common mistake

Reporting a single metric and ignoring the others. Teams often report "our RAG has 0.92 nDCG" without mentioning Recall or Precision, which can hide major problems:

• High nDCG, low Recall: You're ranking the few relevant chunks you find very well, but missing most relevant information entirely. The model can't answer from what it doesn't have.
• High Recall, low Precision: You're finding relevant chunks but buried them in noise. The model might still answer correctly if it's good at filtering, but you're wasting tokens.
• High Precision, low Recall: You're confident in what you retrieve but incomplete. Simple queries work; complex ones fail silently.

Report all four metrics (Recall@k, Precision@k, MRR, nDCG@k) or at least Recall@k and nDCG@k together. They tell complementary stories about retrieval quality.