LLM-as-Judge: Design, Bias, and Calibration

Why LLM-as-judge exists and its cost

Human evaluation is slow and expensive. A team reviewing 1,000 RAG answers at 5 minutes per answer = 83 hours of work, or 2 weeks if one person does it full-time. LLM-as-judge promises to automate this: feed an LLM the question, answer, and a rubric, get a score back in milliseconds at ~$0.001 per call.

But LLMs are not objective. They have biases baked into their training and architecture. An uncalibrated LLM judge is worse than no judge at all — it gives you fake precision and hides its biases behind a number.

The goal of this lesson is: design a judge prompt that minimizes known biases, then measure its accuracy against human labels before trusting it at scale.

Designing the judge prompt: rubric first

The worst judge prompt is a vague one: "Rate this answer 1-10." The model will invent its own criteria mid-evaluation, with no consistency across examples.

A good judge prompt has three parts:

1. Explicit scoring criteria (the rubric)
2. Structured output format (score per criterion, reasoning)
3. Example(s) of what high/medium/low scores look like

Example: a judge for RAG answer quality

def create_rag_judge_prompt(
    question: str,
    answer: str,
    retrieved_context: str,
    rubric: str = None
) -> str:
    """
    Create a judge prompt with explicit rubric and examples.
    """

    if rubric is None:
        rubric = """
        Score the answer on these criteria:

        1. Relevance (0-10): Does the answer directly address the question?
           - 0: Completely off-topic
           - 5: Partially relevant
           - 10: Directly and fully answers the question

        2. Faithfulness (0-10): Is every claim supported by the retrieved context?
           - 0: Multiple contradictions with the context
           - 5: Mostly grounded but 1-2 unsupported claims
           - 10: Every claim is supported or appropriately qualified

        3. Completeness (0-10): Does the answer cover all important aspects?
           - 0: Misses major points
           - 5: Covers main points but misses nuance
           - 10: Comprehensive, addresses all key aspects

        4. Clarity (0-10): Is the answer well-structured and easy to understand?
           - 0: Incoherent or confusing
           - 5: Generally clear with minor issues
           - 10: Clear, well-organized, easy to follow
        """

    prompt = f"""
    You are an expert evaluator of question-answering systems.
    Your task is to grade the provided answer against a rubric.

    QUESTION:
    {question}

    RETRIEVED CONTEXT:
    {retrieved_context}

    ANSWER TO EVALUATE:
    {answer}

    RUBRIC:
    {rubric}

    Provide your evaluation in the following JSON format:
    {{
        "relevance": {{"score": <0-10>, "reasoning": "<brief explanation>"}},
        "faithfulness": {{"score": <0-10>, "reasoning": "<brief explanation>"}},
        "completeness": {{"score": <0-10>, "reasoning": "<brief explanation>"}},
        "clarity": {{"score": <0-10>, "reasoning": "<brief explanation>"}},
        "overall_score": <0-10>,
        "summary": "<1-2 sentence summary of strengths and weaknesses>"
    }}

    Ensure all scores are integers 0-10. Provide reasoning for each score.
    """

    return prompt

Key design choices:

• 0-10 instead of 1-5: Reduces tie-breaking ambiguity. With a 5-point scale, everything bunches in the middle.
• Criteria-specific anchors: Each criterion has explicit 0/5/10 descriptions (not just "rate the relevance"). The model has fewer degrees of freedom.
• JSON output: Structured format lets you parse and store scores programmatically, instead of extracting numbers from free text.
• Reasoning required: Forcing the model to explain its reasoning often makes the score more consistent, and lets you spot when it's confused or using weird criteria.

Three major bias failure modes (and mitigations)

Bias 1: Position bias

The problem: The judge favors whichever answer appears first.

When you pass two answers A and B to a model and ask "which is better?", the model often picks A just because it was mentioned first. This is well-documented in LLM evaluation benchmarks.

Mitigation 1: Randomize order

import json
import random
from anthropic import Anthropic

def evaluate_pair_with_randomization(
    question: str,
    answer_a: str,
    answer_b: str,
    model_id: str = "claude-opus-4-1-20250805"
) -> dict:
    """
    Evaluate two answers by randomizing their presentation order.
    """
    client = Anthropic()

    # Randomly decide which is "first"
    if random.random() < 0.5:
        first_answer, second_answer = answer_a, answer_b
        first_label, second_label = "A", "B"
    else:
        first_answer, second_answer = answer_b, answer_a
        first_label, second_label = "B", "A"

    prompt = f"""
    Evaluate two answers to the following question:

    QUESTION: {question}

    FIRST ANSWER ({first_label}):
    {first_answer}

    SECOND ANSWER ({second_label}):
    {second_answer}

    Which answer is better overall? Score both 0-10 on relevance, faithfulness, and completeness.
    Provide your scores as JSON.
    """

    response = client.messages.create(
        model=model_id,
        max_tokens=500,
        messages=[{"role": "user", "content": prompt}]
    )

    # Parse JSON from response
    scores = json.loads(response.content[0].text)

    # Map back to original labels (A, B) if we randomized
    if first_label == "B":
        scores["A"] = scores.pop("B")
        scores["B"] = scores.pop("A")

    return scores

Mitigation 2: Evaluate in isolation

Even better: don't do pairwise comparisons. Evaluate each answer independently against a rubric, then compare their scores afterward. This removes the direct preference signal entirely.

def evaluate_independently(
    question: str,
    answer_a: str,
    answer_b: str
) -> dict:
    """
    Evaluate each answer independently, then compare scores.
    """
    score_a = evaluate_single_answer(question, answer_a)
    score_b = evaluate_single_answer(question, answer_b)

    # Now compare scores, not answers
    # This avoids the "which is first" bias
    return {
        "answer_a_score": score_a,
        "answer_b_score": score_b,
        "winner": "A" if score_a > score_b else "B"
    }

Bias 2: Verbosity bias

The problem: The judge gives higher scores to longer answers, even when they're not better.

# Example: a short, correct answer vs. a longer, padded answer

short_answer = "The refund window is 14 days from purchase."
long_answer = """
When you make a purchase on our platform, you have a certain amount of time to request a refund.
This timeframe is important to understand for your financial planning. Specifically, for our standard
digital products, the refund window is 14 days from the date of your purchase. This is our standard
refund window. It's important to note that you have up to 14 days. After 14 days have passed, your
purchase is final and no refunds are available.
"""

# A verbosity-biased judge often scores the long answer higher (0.75)
# vs the short answer (0.65), even though they say the same thing.

Mitigation: Explicitly penalize or normalize for length

def evaluate_with_length_adjustment(
    question: str,
    answer: str,
    token_limit: int = 300
) -> dict:
    """
    Evaluate answer, then adjust score if it exceeds reasonable length.
    """

    # Get base score
    base_score = evaluate_single_answer(question, answer)

    # Count approximate tokens
    token_count = len(answer.split())

    # If answer is unnecessarily long, penalize
    if token_count > token_limit:
        verbosity_penalty = 0.1 * (token_count - token_limit) / token_limit
        adjusted_score = max(0, base_score - verbosity_penalty)
    else:
        adjusted_score = base_score

    return {
        "base_score": base_score,
        "token_count": token_count,
        "verbosity_penalty": verbosity_penalty if token_count > token_limit else 0,
        "final_score": adjusted_score
    }

Better mitigation: Include length in the rubric

rubric = """
...

5. Conciseness (0-10): Is the answer concise without losing important details?
   - 0: Unnecessarily verbose or rambling
   - 5: Appropriate length
   - 10: Every sentence adds value

Include conciseness in your overall score.
"""

Bias 3: Self-preference bias

The problem: A model rates outputs from its own family more highly than outputs from competitors.

If you use Claude to judge Claude's answers, it might be softer than if you use Claude to judge GPT-4's answers. This isn't intentional bias in training, but a natural side effect of learning from examples.

Mitigation: Use a different model family for judging

def evaluate_with_neutral_judge(
    question: str,
    answer: str,
    model_id: str = "gpt-4o"  # Use GPT to judge Claude (or vice versa)
) -> dict:
    """
    Evaluate using a model from a different family.
    """

    prompt = create_rag_judge_prompt(question, answer, retrieved_context)

    # Use OpenAI's API
    response = openai.ChatCompletion.create(
        model=model_id,  # e.g., "gpt-4o" judging Claude's answer
        messages=[{"role": "user", "content": prompt}]
    )

    return json.loads(response['choices'][0]['message']['content'])

Trade-off: This adds cost and latency (using a second vendor's API). But the reduction in bias is worth it for high-stakes evaluation.

Calibration: comparing judge scores to human labels

An uncalibrated judge is dangerous because it feels objective. You need to measure its accuracy before deploying it.

Calibration process:

1. Pick ~50 examples from your golden set (question, answer, retrieved context)
2. Have 2-3 humans score each on the same rubric
3. Compute human inter-rater agreement (Cohen's kappa or Fleiss' kappa)
4. Run the LLM judge on the same 50 examples
5. Compare LLM scores to human consensus
6. Compute correlation (Spearman's rho or Pearson's r)

from scipy.stats import spearmanr, pearsonr
import numpy as np

def calibrate_judge(
    examples: List[dict],  # [{"question": "", "answer": "", "context": ""}, ...]
    human_labels: List[List[float]],  # shape: (n_examples, n_raters, n_criteria)
    judge_function,  # function that evaluates and returns dict with scores
    criteria: List[str] = ["relevance", "faithfulness", "completeness", "clarity"]
) -> dict:
    """
    Calibrate an LLM judge against human gold labels.
    """

    # Compute human consensus (mean of raters)
    human_consensus = np.mean(human_labels, axis=1)  # shape: (n_examples, n_criteria)

    # Run judge on all examples
    judge_scores = []
    for example in examples:
        result = judge_function(
            example["question"],
            example["answer"],
            example["context"]
        )
        # Extract scores for each criterion
        scores = [result[criterion]["score"] for criterion in criteria]
        judge_scores.append(scores)

    judge_scores = np.array(judge_scores)  # shape: (n_examples, n_criteria)

    # Compute correlation for each criterion
    correlations = {}
    for i, criterion in enumerate(criteria):
        human_col = human_consensus[:, i]
        judge_col = judge_scores[:, i]
        corr, pval = spearmanr(human_col, judge_col)
        correlations[criterion] = {"spearman_r": corr, "p_value": pval}

    # Compute overall correlation
    human_overall = np.mean(human_consensus, axis=1)
    judge_overall = np.mean(judge_scores, axis=1)
    overall_corr, overall_pval = spearmanr(human_overall, judge_overall)

    return {
        "per_criterion": correlations,
        "overall_correlation": {
            "spearman_r": overall_corr,
            "p_value": overall_pval
        },
        "judge_scores": judge_scores,
        "human_consensus": human_consensus
    }

Interpreting results:

| Correlation | Interpretation | Action | |---|---|---| | r > 0.8 | Excellent agreement | Safe to deploy | | 0.6 < r ≤ 0.8 | Good but not perfect | Deploy with sampling audits | | 0.4 < r ≤ 0.6 | Moderate; don't trust fully | Revise rubric and retrain | | r ≤ 0.4 | Poor agreement | Don't use; major redesign needed |

If correlation is low, investigate:

1. Is the rubric ambiguous? Ask: "Do humans disagree with each other?" If human inter-rater agreement is also low (kappa < 0.5), the rubric needs clarification, not the judge.
2. Is the judge misunderstanding criteria? Read the judge's reasoning. Does it use different definitions than the rubric? If so, add examples to the prompt.
3. Is the judge biased toward a specific failure mode? Plot judge scores vs. human scores by criterion. If one criterion is systematically off, that criterion needs adjustment.

Real deployment: monitoring judge drift

Even if a judge is well-calibrated at deployment time, it can drift over time:

• Model updates: If the underlying model changes (e.g., Claude 3.5 Sonnet → Claude 4), the judge's behavior may shift.
• Distribution shift: If your answers change (e.g., you switch to a different RAG architecture), the judge's performance may degrade.

Defense: periodic recalibration

def monitor_judge_performance(
    judge_function,
    recent_examples: List[dict],  # new examples from the past week
    human_labels: List[List[float]],  # human reviews of those examples
    previous_correlation: float  # e.g., 0.82 from initial calibration
):
    """
    Check if judge has drifted significantly from initial calibration.
    """

    result = calibrate_judge(recent_examples, human_labels, judge_function)
    current_correlation = result["overall_correlation"]["spearman_r"]

    drift = previous_correlation - current_correlation

    if drift > 0.05:  # threshold: 5% drop
        print(f"ALERT: Judge drift detected. Correlation dropped from {previous_correlation} to {current_correlation}")
        return "needs_recalibration"
    else:
        print(f"Judge performance stable. Correlation: {current_correlation}")
        return "ok"

Run this check monthly on a sample of 30-50 new examples that humans have labeled. If drift exceeds your threshold, re-calibrate or adjust the judge prompt.

Example: a complete judge setup for a customer support RAG

def create_support_judge():
    """
    Full, production-ready judge setup.
    """

    rubric = """
    Score the customer support answer on:

    1. Relevance (0-10): Does it answer the customer's question?
    2. Accuracy (0-10): Is every fact correct according to company policy?
    3. Clarity (0-10): Is it easy for a non-technical customer to understand?
    4. Tone (0-10): Is it professional and helpful?

    Anchor each score:
    - 0: Fails completely
    - 5: Acceptable but with significant issues
    - 10: Excellent
    """

    def judge_answer(question: str, answer: str, context: str) -> dict:
        """
        Judge a single answer.
        """
        prompt = f"""
        You are grading customer support answers. Be consistent and fair.

        RUBRIC:
        {rubric}

        CUSTOMER QUESTION:
        {question}

        COMPANY CONTEXT (policy, guidelines):
        {context}

        ANSWER TO GRADE:
        {answer}

        Provide scores as JSON:
        {{
            "relevance": {{"score": <0-10>, "reasoning": "..."}},
            "accuracy": {{"score": <0-10>, "reasoning": "..."}},
            "clarity": {{"score": <0-10>, "reasoning": "..."}},
            "tone": {{"score": <0-10>, "reasoning": "..."}},
            "overall_score": <0-10>
        }}
        """

        # Use GPT to judge Claude outputs (neutral judge)
        response = openai.ChatCompletion.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
            temperature=0.0  # deterministic
        )

        return json.loads(response['choices'][0]['message']['content'])

    # Calibrate on 50 golden examples
    calibration_result = calibrate_judge(
        examples=load_golden_examples(50),
        human_labels=load_human_labels(50),
        judge_function=judge_answer
    )

    print(f"Judge calibration complete. Correlation: {calibration_result['overall_correlation']['spearman_r']:.2f}")

    return judge_answer, calibration_result

Common mistake

Deploying a judge without calibration. Teams often think: "I'll use GPT-4 to grade answers; it's smart and probably accurate." Then they evaluate their whole pipeline and get a score, without ever checking if the judge's scores match what humans would give.

Three months later, they notice the judge is giving consistently high scores, but user satisfaction is dropping. They realize they never calibrated. The "accurate" judge was actually giving false positives.

Always calibrate on at least 50 examples before trusting a judge at scale. The calibration is the only way you know whether disagreement between the judge and humans means the rubric is broken or the cases are genuinely ambiguous.