Generation Quality and Faithfulness Metrics

The generation quality problem: relevance ≠ faithfulness

You've retrieved the right chunks. Now the LLM needs to generate an answer grounded in those chunks, not hallucinate.

Two separate dimensions define answer quality:

• Relevance: Does the answer address the user's question? (Even if it's wrong, it's at least on-topic.)
• Faithfulness: Is every claim in the answer supported by or consistent with the retrieved evidence?

A system can score high on both, one, or neither:

| | Relevant | Irrelevant | |---|---|---| | Faithful | ✓ Ideal: right answer, grounded | ✓ Good enough: answer is unfocused but not wrong | | Hallucinating | ✗ Worst: confident wrong answer (misleads) | ✗ Useless: wrong answer to wrong question |

Example: Query: "What's the refund deadline for digital products?"

Good answer (relevant + faithful): "According to our policy, digital products have a 14-day refund window from purchase. You can request a refund through the Account Settings menu."

Bad answer (relevant + hallucinating): "According to our policy, digital products have a 30-day refund window and you can transfer the product to another account within that time." (The "30-day" and "transfer" claims were never in the retrieved chunks.)

Off-topic but harmless (irrelevant + faithful): "Digital products are downloaded through our cloud platform. The refund deadline is 14 days." (Answers the question but includes unnecessary detail.)

Worst answer (irrelevant + hallucinating): "Digital products can be refunded for up to a year, and you get 150% of the purchase price back as store credit." (Both wrong and unrelated.)

Scoring only "is this answer good?" misses the difference between a minor irrelevance and a major hallucination. Claim-level scoring keeps that distinction visible.

Understanding hallucinations in RAG

Hallucinations in RAG fall into three categories, each requiring different detection strategies:

1. Intrinsic hallucinations: Claims that contradict the retrieved context.

• Example: Chunks say "refund within 14 days," answer says "refund within 30 days."
• Detectability: High. Easy pattern matching or semantic matching.

2. Extrinsic hallucinations: Claims that aren't contradicted but aren't supported either.

• Example: Chunks mention "14-day refund window," answer adds "this is industry standard" (may be true in training data, but not in retrieved chunks).
• Detectability: Medium. Requires checking that claims are grounded in retrieved evidence specifically, not general knowledge.

3. Reasoning errors: Valid claims but misapplied logic.

• Example: Chunks say "refund within 14 days; no refunds on digital products." Answer says "I can refund your digital product in 14 days." (Contradicts the logic, even if individual facts are stated.)
• Detectability: Low. Requires understanding multi-step reasoning.

Most automated systems focus on 1 and 2; reasoning errors need human review or very sophisticated LLM grading.

Claim extraction and scoring framework

The core idea: break an answer into atomic claims, then check each claim against evidence.

from dataclasses import dataclass
from typing import List, Tuple

@dataclass
class Claim:
    text: str                    # e.g., "Refunds are issued within 5 business days"
    support_status: str          # "supported", "unsupported", "contradicted"
    evidence_chunk_ids: List[int] = None  # which chunk(s) support it

def extract_claims_naive(answer_text: str) -> List[str]:
    """
    Naive claim extraction: split by sentences.
    Production systems use a fine-tuned model or LLM call for better accuracy.
    """
    import re
    sentences = re.split(r'(?<=[.!?])\s+', answer_text)
    return [s.strip() for s in sentences if len(s.strip()) > 10]

def match_claim_to_evidence(claim: str, evidence_chunks: List[str]) -> Tuple[str, List[int]]:
    """
    Check if a claim is supported by evidence chunks.
    Returns: (status, chunk_indices)
    """
    from difflib import SequenceMatcher

    statuses = []

    for i, chunk in enumerate(evidence_chunks):
        # Simple substring match (production: use embeddings or LLM)
        if claim.lower() in chunk.lower():
            statuses.append(("supported", i))
        # Check for contradiction (simple: opposite keywords)
        elif contains_negation_of_claim(claim, chunk):
            statuses.append(("contradicted", i))

    if any(status == "supported" for status, _ in statuses):
        return "supported", [i for status, i in statuses if status == "supported"]
    elif any(status == "contradicted" for status, _ in statuses):
        return "contradicted", [i for status, i in statuses if status == "contradicted"]
    else:
        return "unsupported", []

def contains_negation_of_claim(claim: str, chunk: str) -> bool:
    """
    Heuristic: detect if chunk negates the claim.
    E.g., claim = "refunds allowed", chunk contains "no refunds" or "refunds not allowed".
    """
    import re
    # Extract key noun phrase from claim
    words = claim.lower().split()
    main_noun = words[-1] if words else ""

    # Check for negations in chunk
    negation_patterns = [f"no {main_noun}", f"{main_noun} not", f"not {main_noun}"]
    for pattern in negation_patterns:
        if pattern in chunk.lower():
            return True

    return False

def evaluate_answer_faithfulness(
    answer_text: str,
    evidence_chunks: List[str]
) -> dict:
    """
    Full pipeline: extract claims, match each to evidence, return summary.
    """
    claims = extract_claims_naive(answer_text)

    claim_results = []
    for claim in claims:
        status, chunk_ids = match_claim_to_evidence(claim, evidence_chunks)
        claim_results.append(Claim(text=claim, support_status=status, evidence_chunk_ids=chunk_ids))

    # Summary metrics
    supported_count = sum(1 for c in claim_results if c.support_status == "supported")
    contradicted_count = sum(1 for c in claim_results if c.support_status == "contradicted")
    unsupported_count = sum(1 for c in claim_results if c.support_status == "unsupported")

    faithfulness_score = supported_count / len(claim_results) if claim_results else 1.0

    return {
        "total_claims": len(claim_results),
        "supported": supported_count,
        "unsupported": unsupported_count,
        "contradicted": contradicted_count,
        "faithfulness_score": faithfulness_score,  # 0 to 1, higher is better
        "claims": claim_results
    }

Running this on an example:

answer = """
Refunds for digital products are processed within 5 business days.
You can request a refund through the Account Settings menu.
Once approved, the refund is issued directly to your original payment method.
Digital products are typically non-refundable after 14 days, but exceptions can be made.
"""

evidence = [
    "Refund Policy: Digital products have a 14-day refund window from purchase.",
    "To request a refund, navigate to Account Settings > Purchases > Refund Request.",
    "Approved refunds are issued to the original payment method within 5-7 business days.",
    "Exceptions to the refund policy require manager approval and are rare."
]

result = evaluate_answer_faithfulness(answer, evidence)

# Output:
# {
#     "total_claims": 5,
#     "supported": 3,       # claims 2, 3, 5 match evidence
#     "unsupported": 1,     # claim 4 "14 days" is mentioned but in different context
#     "contradicted": 1,    # claim 1 says "5 business days" but evidence says "5-7"
#     "faithfulness_score": 0.6,
#     "claims": [
#         Claim(text="Refunds for digital products are processed within 5 business days.",
#               support_status="contradicted", evidence_chunk_ids=[2]),
#         Claim(text="You can request a refund through the Account Settings menu.",
#               support_status="supported", evidence_chunk_ids=[1]),
#         ...
#     ]
# }

Key insights from this example:

• Claim 1 is technically contradicted: evidence says "5-7 days," not exactly "5 days"
• Claim 5 about exceptions is unsupported (vague, not clearly grounded)
• Overall faithfulness = 0.6, below ideal, but not the "hallucinating garbage" end of the spectrum

Automated faithfulness scoring with Ragas

Rather than building from scratch, teams often use frameworks like Ragas, which provides a faithfulness metric that automates claim extraction and scoring.

# Install Ragas
pip install ragas

from ragas import evaluate
from ragas.metrics import faithfulness
from datasets import Dataset

# Prepare evaluation data in Ragas format
eval_data = {
    "question": ["What is the refund policy for digital products?"],
    "answer": ["Refunds are processed within 5 business days. You can request a refund through Account Settings."],
    "contexts": [[  # list of retrieved chunks, in order
        "Refund Policy: Digital products have a 14-day refund window from purchase.",
        "To request a refund, navigate to Account Settings > Purchases > Refund Request."
    ]]
}

dataset = Dataset.from_dict(eval_data)

# Run Ragas evaluation
results = evaluate(dataset, metrics=[faithfulness])

print(results)
# Output:
# {
#     'faithfulness': 0.92
# }

What Ragas faithfulness does under the hood:

1. Uses an LLM (e.g., GPT-4 or Claude) to extract claims from the answer
2. Grades each claim against the retrieved context using a separate LLM call with an explicit rubric
3. Scores each claim on a 0-1 scale (0 = unsupported/contradicted, 1 = supported)
4. Averages claim scores into a faithfulness metric for the whole answer

Advantages:

• More accurate than keyword matching; understands paraphrases and synonyms
• Uses an explicit rubric, reducing subjective bias in scoring
• Produces detailed claim-level feedback (not just a single number)

Disadvantages:

• Requires API calls (cost, latency)
• The LLM grader can have its own biases and failure modes
• Sensitive to prompt wording in the claim-extraction LLM call
• May not catch subtle reasoning errors or misapplications of facts

Combining faithfulness with relevance

A complete generation evaluation scores both:

def evaluate_generation_quality(
    question: str,
    answer: str,
    retrieved_context: List[str]
) -> dict:
    """
    Full generation evaluation: relevance + faithfulness.
    """

    # Metric 1: Faithfulness (is the answer grounded?)
    faithfulness_result = evaluate_answer_faithfulness(answer, retrieved_context)
    faithfulness_score = faithfulness_result["faithfulness_score"]

    # Metric 2: Relevance (does it answer the question?)
    # Use an LLM or embeddings to judge relevance
    relevance_score = grade_answer_relevance(question, answer)
    # (0 to 1, higher = more directly answers the question)

    # Metric 3: Answer quality (combines both)
    answer_quality = (faithfulness_score * 0.6) + (relevance_score * 0.4)
    # Example weighting: faithfulness is twice as important as relevance
    # (adjust based on your use case)

    return {
        "faithfulness": faithfulness_score,
        "relevance": relevance_score,
        "overall_quality": answer_quality,
        "faithfulness_details": faithfulness_result
    }

def grade_answer_relevance(question: str, answer: str) -> float:
    """
    LLM-based relevance grading.
    """
    from anthropic import Anthropic

    client = Anthropic()

    prompt = f"""
    Question: {question}

    Answer: {answer}

    Does this answer address the question? Score 0-10 where:
    - 0: completely off-topic
    - 5: partially addresses the question
    - 10: directly and fully answers the question

    Respond with a single number.
    """

    response = client.messages.create(
        model="claude-opus-4-1-20250805",
        max_tokens=10,
        messages=[{"role": "user", "content": prompt}]
    )

    score = int(response.content[0].text.strip()) / 10  # normalize to 0-1
    return score

Setting thresholds:

• Faithfulness < 0.7: Unacceptable. Likely contains hallucinations. Retrain or adjust retrieval.
• Relevance < 0.6: Answer misses the question. Check if retrieval got the right context, or prompt is unclear.
• Overall quality < 0.65: Not ready for users. Investigate both layers.

Real case: hallucination detection at a financial services company

A RAG system answers "What's my current interest rate on my savings account?"

Question: "What's my current interest rate?"

Correct answer (from knowledge base): "Your current savings account interest rate is 4.5% APY, updated as of July 1, 2024."

Hallucinated answer (what the model generated): "Your current savings account interest rate is 5.2% APY, compounded daily."

Retrieved context (what was in the chunks):

• "Standard savings accounts earn 4.5% APY"
• "Interest is compounded daily for all accounts"
• "Rates are updated quarterly, last on July 1, 2024"

Claim-level analysis:

1. "interest rate is 5.2% APY" — CONTRADICTED (chunks say 4.5%)
2. "compounded daily" — SUPPORTED (chunk 2 explicitly states this)

Faithfulness score = 1/2 = 0.50 (one out of two claims is supported).

What went wrong: The model "knew" from training data that high-yield savings rates can be in the 5% range. It hallucinated a plausible-looking number instead of grounding itself in the retrieved evidence. A naive relevance check would have missed this (the answer is about interest rates, so it's "relevant"). Only claim-level faithfulness scoring caught the error.

How to prevent it: Strengthen the system prompt: "Answer based ONLY on the provided context. If the exact rate is not in the context, say so instead of guessing."

Common mistake

Relying on a single "hallucination: yes/no" binary score. Teams often ask: "Is this answer hallucinated?" and expect a simple true/false answer. But hallucination is a spectrum, not a binary:

• No hallucination: Every claim is directly supported.
• Minor hallucination: One unsupported side detail, but the main answer is grounded.
• Major hallucination: Multiple contradictions or central claims are unsupported.
• Complete hallucination: The entire answer contradicts evidence or is unrelated.

Report faithfulness as a continuous score (0-1) with claim-level breakdowns, not as a binary yes/no. This lets you catch and prioritize the most serious hallucinations while tolerating minor, harmless extraneous details.

Also, don't let faithfulness checks be your only defense. An answer can be 100% faithful to the wrong evidence (if retrieval failed silently). Always evaluate retrieval metrics and generation metrics together.