Human Evaluation: Rubrics, Sampling, and Calibration

Why human evaluation matters

Automated metrics (recall, faithfulness, LLM-as-judge) are cheap but noisy. Human evaluation is expensive but honest — humans can catch subtle failures that metrics miss and provide qualitative feedback ("this answer is technically correct but confusing").

Most production RAG systems use both: humans grade a sample (10-20% of data), LLM judges scale to the rest, and the human-judge correlation measures whether automation is trustworthy.

Designing a rubric humans can apply consistently

A bad rubric is vague: "Is this answer good?" A good rubric is specific and measurable: "Does every claim have a source listed?"

The difference is the difference between a kappa (inter-rater agreement) of 0.3 (poor) and 0.8 (excellent).

Rubric design checklist:

1. Use concrete criteria, not subjective ones

   • ❌ Bad: "Is the tone appropriate?" (different raters have different notions of appropriate)
   • ✓ Good: "Does the answer address the user's question without editorializing?" (binary, observable)

2. Provide decision trees or decision rules, not just descriptions

   • ❌ Bad: "Rate accuracy on a 1-5 scale."
   • ✓ Good: "Is every numeric claim verifiable against the retrieved context? (1 = no claims verified, 5 = all verified)"

3. Include examples of borderline cases

   • ❌ Bad: Just the rubric text
   • ✓ Good: Rubric text + 2 examples of Score 3 (middle), 1 of Score 5, 1 of Score 1

4. Define the "scope" of review narrowly

   • ❌ Bad: "Is this a good answer?" (what does "good" mean?)
   • ✓ Good: "Does the answer directly address the question the user asked, without tangents?" (narrow, focused)

Example rubric for RAG evaluation:

RUBRIC_RAG_ANSWER_QUALITY = """
SCORE ACCURACY: Does every factual claim have a source in the retrieved context?

1 = None of the claims are supported by the context (most claims are hallucinations)
2 = Some claims are supported, but there are multiple unsupported or contradicted claims
3 = Most claims are supported (1-2 unsupported side remarks are okay)
4 = Nearly all claims are supported; only very minor quibbles possible
5 = Every claim is either supported or appropriately qualified ("based on context" or "outside provided context")

EXAMPLES:

Example A (retrieved context mentions "14-day refund window" and "refunds processed within 5 days"):
  Answer: "You can get a refund within 14 days of purchase, processed in 5 business days."
  SCORE: 5 (both facts are in context, appropriately stated)

Example B (same context):
  Answer: "You can get a refund within 14 days, but not if you used the product. It's processed in 3 days."
  SCORE: 2 (first claim is supported; "3 days" contradicts "5 days"; "no refund if used" is unsupported)

Example C (context has nothing about whether product was used):
  Answer: "Refunds are available within 14 days. Most customers receive them in 5 days."
  SCORE: 4 ("most customers" is a mild generalization not in context, but main facts are correct)

---

SCORE COMPLETENESS: Does the answer cover the main points the user was asking about?

1 = Misses major points; answer is incomplete
2 = Covers some main points but misses others
3 = Covers all main points; may miss minor details
4 = Comprehensive; covers main points and relevant details
5 = Excellent; anticipates follow-up questions and provides context

EXAMPLES:

Query: "How do I cancel my subscription?"
Answer: "Go to Account Settings > Subscription > Cancel."
SCORE: 3 (covers the main action but doesn't mention what happens to prepaid balance, refunds, etc.)

Query: "How do I cancel my subscription?"
Answer: "Go to Account Settings > Subscription > Cancel. Any unused prepaid time will be refunded to your account. Cancellation is immediate."
SCORE: 4 (covers main action and important follow-up details)
"""

def apply_rubric(answer: str, question: str, context: str, rater_id: str) -> dict:
    """
    A human rater applies the rubric to a single answer.
    Returns their scores.
    """
    return {
        "rater_id": rater_id,
        "accuracy_score": None,  # rater fills this in
        "completeness_score": None,  # rater fills this in
        "reasoning": None,  # rater explains their choice
    }

Sampling strategies: which examples to have humans grade

You can't afford to have humans review every answer. Choose strategically.

Random sampling

Use when: You want a representative sample that reflects overall system performance.

import random

def random_sample(examples: List[dict], sample_size: int = 100) -> List[dict]:
    """
    Pick `sample_size` examples uniformly at random.
    """
    return random.sample(examples, k=min(sample_size, len(examples)))

Pros: No bias; sample matches your overall distribution. Cons: Might miss edge cases (rare but important scenarios). Sample size rule: For a 95% confidence interval, sample_size ≈ 400 / (margin_of_error)^2. For a 5% error margin, sample ~1,600. For 10%, sample ~400.

Stratified sampling

Use when: You want to ensure representation of important subgroups (e.g., query types, domains).

from collections import defaultdict

def stratified_sample(
    examples: List[dict],
    stratify_by: str,  # e.g., "domain" or "query_type"
    sample_size: int = 100
) -> List[dict]:
    """
    Sample proportionally from each stratum.
    """
    # Group by stratum
    strata = defaultdict(list)
    for example in examples:
        strata[example[stratify_by]].append(example)

    # Sample from each stratum proportionally
    sample = []
    for stratum_name, stratum_examples in strata.items():
        stratum_sample_size = int(sample_size * len(stratum_examples) / len(examples))
        sample.extend(random.sample(stratum_examples, k=stratum_sample_size))

    return sample

# Example: ensure you review a balanced mix of query types
sample = stratified_sample(examples, "query_type", sample_size=100)
# e.g., 30 billing queries, 25 refund, 25 account, 20 technical

Pros: Ensures balanced representation across important categories. Cons: Requires defining strata upfront; adds complexity.

Targeted sampling (error mining)

Use when: You want to find and fix the system's worst failures.

def find_hard_cases(
    examples: List[dict],
    metric_function,  # e.g., compute_nDCG
    percentile: int = 10  # bottom 10%
) -> List[dict]:
    """
    Find examples where the system performed worst.
    """
    scores = [(example, metric_function(example)) for example in examples]
    scores.sort(key=lambda x: x[1])  # sort by score ascending

    # Return the worst `percentile`%
    cutoff_idx = int(len(scores) * percentile / 100)
    return [example for example, _ in scores[:cutoff_idx]]

# Example: review the 50 worst-scoring RAG queries
hard_cases = find_hard_cases(examples, compute_nDCG, percentile=10)

Pros: Focuses effort on the biggest problems; maximizes learning per review. Cons: Biased sample; doesn't reflect overall performance. Use case: After an initial random sample, use targeted sampling to understand failure modes.

Mixed approach (best practice)

def sample_for_evaluation(
    examples: List[dict],
    total_sample_size: int = 100
) -> dict:
    """
    Combine strategies for best coverage.
    """
    # 50% random (representative)
    random_sample_list = random_sample(examples, sample_size=50)

    # 30% stratified by domain
    stratified_sample_list = stratified_sample(examples, "domain", sample_size=30)

    # 20% hard cases (focus on failures)
    hard_cases_sample_list = find_hard_cases(examples, percentile=5)[:20]

    # Deduplicate and combine
    all_samples = random_sample_list + stratified_sample_list + hard_cases_sample_list
    seen = set()
    unique_samples = []
    for sample in all_samples:
        if sample["id"] not in seen:
            unique_samples.append(sample)
            seen.add(sample["id"])

    return {
        "random": random_sample_list,
        "stratified": stratified_sample_list,
        "hard_cases": hard_cases_sample_list,
        "all": unique_samples
    }

Inter-rater agreement: detecting rubric problems

When you have 2+ raters score the same examples, disagreement signals one of two things:

1. Rubric is unclear → raters use different criteria (bad)
2. Cases are genuinely ambiguous → raters disagree because the situation is inherently unclear (okay, fix the rubric)

Cohen's kappa measures this:

$$\kappa = \frac$$

where:

• $p_o$ = proportion of examples where raters agree
• $p_e$ = proportion of agreement expected by chance

Kappa ranges from -1 to 1:

• 0.8+ = Excellent agreement (rubric is clear)
• 0.6-0.8 = Moderate agreement (acceptable, but refine rubric)
• 0.4-0.6 = Fair agreement (rubric needs work)
• <0.4 = Poor agreement (rubric is too vague; don't use)

from sklearn.metrics import cohen_kappa_score

def compute_inter_rater_agreement(
    rater_1_scores: List[int],
    rater_2_scores: List[int]
) -> float:
    """
    Compute Cohen's kappa for two raters.
    """
    kappa = cohen_kappa_score(rater_1_scores, rater_2_scores)
    return kappa

# Example:
rater_1_accuracy_scores = [5, 4, 3, 5, 2, 4, 5, 3]  # 8 examples
rater_2_accuracy_scores = [5, 4, 2, 5, 2, 3, 5, 4]  # rater 2's scores

kappa = compute_inter_rater_agreement(rater_1_accuracy_scores, rater_2_accuracy_scores)
print(f"Cohen's kappa: {kappa:.2f}")
# If kappa < 0.6, the rubric needs clarification before scaling to more raters

What to do if kappa is low:

1. Read the disagreements. Look at examples where raters disagreed most.
   • Example: Both raters gave score 3 or higher, but rater 1 said 5 and rater 2 said 3. → Rubric is unclear about the difference between "good" and "excellent."
2. Update the rubric with more specific anchors. E.g., add an example of a score-3 case.
3. Re-test on the same examples. Kappa should improve.
4. Only then scale to more raters.

Reconciliation: what to do with disagreements

When raters disagree, you have options:

Option 1: Majority vote If 3 raters gave [5, 5, 4], score = 5 (majority). Simple, but loses information.

Option 2: Mean score Score = (5 + 5 + 4) / 3 = 4.67. Preserves nuance, but may be ambiguous (what does 4.67 mean?).

Option 3: Discussion and consensus Raters discuss the borderline cases and come to agreement. Slow, but highest quality. Use for calibration data only.

def reconcile_scores(
    scores_by_rater: dict,  # {"rater_1": [5, 4, ...], "rater_2": [5, 3, ...], ...}
    method: str = "mean"  # "majority", "mean", or "discussion"
) -> List[float]:
    """
    Combine rater scores into final scores.
    """
    n_examples = len(list(scores_by_rater.values())[0])
    final_scores = []

    for example_idx in range(n_examples):
        scores = [scores_by_rater[rater][example_idx] for rater in scores_by_rater]

        if method == "mean":
            final_scores.append(sum(scores) / len(scores))
        elif method == "majority":
            # Round mean to nearest integer
            final_scores.append(round(sum(scores) / len(scores)))
        elif method == "discussion":
            # Placeholder; actual discussion is manual
            final_scores.append(sum(scores) / len(scores))

    return final_scores

For golden-set / calibration data, use discussion: have raters talk through disagreements and reach consensus. This ensures your ground truth is as clear as possible.

Feeding human labels into LLM-judge calibration

Once you have human-scored examples, you compare them to your LLM judge:

def calibrate_judge_with_human_labels(
    examples: List[dict],  # [{"question": "", "answer": "", "context": ""}, ...]
    human_scores: List[List[float]],  # shape: (n_examples, n_raters, n_criteria)
    judge_function,  # LLM judge
    criteria: List[str] = ["accuracy", "completeness"]
):
    """
    Compare judge scores to human consensus.
    """
    from scipy.stats import spearmanr

    # Human consensus (mean across raters)
    human_consensus = []
    for example_scores in human_scores:
        example_consensus = sum(example_scores) / len(example_scores)
        human_consensus.append(example_consensus)

    # Judge scores
    judge_scores = []
    for example in examples:
        result = judge_function(example["question"], example["answer"], example["context"])
        # Extract overall score
        judge_scores.append(result["overall_score"])

    # Correlation
    corr, pval = spearmanr(human_consensus, judge_scores)

    print(f"Judge vs. human correlation: r = {corr:.2f}, p = {pval:.4f}")

    if corr > 0.7:
        print("✓ Judge is well-calibrated. Safe to use at scale.")
    elif corr > 0.5:
        print("⚠ Judge is decent but not great. Audit samples regularly.")
    else:
        print("✗ Judge disagrees with humans. Revise prompt or rubric.")

    return {"correlation": corr, "p_value": pval}

Interpreting low correlation:

• If humans also disagreed with each other (low inter-rater agreement), the issue is the rubric, not the judge.
• If humans agreed but the judge disagreed, the judge prompt needs adjustment.

A real example: customer support evaluation at scale

Scenario: A support team deployed a RAG-backed chatbot. They want to evaluate 1,000 conversations, but can only afford to have humans review a sample.

Step 1: Design rubric (week 1)

RUBRIC_SUPPORT_ANSWER_QUALITY

ACCURACY (is it correct?):
1 = Factually wrong or contradicts policy
2 = Mostly right but with errors
3 = Correct
4 = Correct and with helpful context
5 = Correct, helpful, and anticipates follow-ups

HELPFULNESS (does it solve the problem?):
1 = Doesn't address the question
2 = Addresses part of the question
3 = Addresses the full question
4 = Addresses question + provides next steps
5 = Addresses question, provides next steps, and offers alternatives

(Include examples for scores 1, 3, 5 for each criterion...)

Step 2: Pilot with 50 examples (week 2)

• 3 human raters score 50 examples each
• Compute inter-rater agreement (Cohen's kappa)
• Kappa = 0.68 (moderate; revise rubric)

Step 3: Refine rubric, re-test (week 2.5)

Raters found "accuracy" ambiguous: "correct according to who?" Revise to: "Does every fact match the company's documented policies?"

Retest on same 50: kappa = 0.82 (excellent!)

Step 4: Sample 200 examples stratified by issue type (week 3)

• 50 billing issues (proportional representation)
• 50 technical issues
• 50 account/password issues
• 50 product/refund issues

Have 2 raters score each. Average their scores.

Step 5: Calibrate judge (week 4)

• Run LLM judge on same 200 examples
• Compare judge scores to human consensus
• Correlation = 0.76 (good; safe to deploy)

Step 6: Deploy with monitoring (week 5+)

• Judge evaluates all 1,000 conversations automatically
• Monthly, randomly sample 50 new conversations and have humans score them
• Track if judge-human correlation stays > 0.7
• If drops below 0.65, re-calibrate

Common mistake

Assuming one rater is "the truth." Teams often have one senior person score a few examples, then trust the LLM judge to match that one person's scores. But one person's score is noisy; multiple raters averaging is much more reliable.

Also, don't skip inter-rater agreement. "I think my rubric is clear" doesn't count. Measure agreement, and if it's below 0.7, refine. This is the only way to know if your rubric actually works.

Finally, don't conflate "humans disagree" with "rubric is bad." Some cases are genuinely ambiguous (a borderline answer that could reasonably be scored 3 or 4). The rubric is working correctly if raters disagree on ambiguous cases but agree on clear cases. If they disagree on clear cases, the rubric needs work.