Evaluating and Testing Agents

Why testing agents is different

Testing a traditional software function is straightforward: given input X, verify output Y. But agents are non-deterministic.

• Same input, different outputs: Ask the same question twice, and the agent may search differently or reach different conclusions.
• Success is nuanced: Is an answer that's mostly correct but incomplete a pass or fail?
• Hidden costs: An agent might produce the right answer but use 3 API calls when 1 would suffice.

Agent evaluation needs metrics that capture this complexity.

Evaluation framework overview

Before diving into code, here's a high-level view of how metrics fit into the development lifecycle:

| Phase | Key Metrics | Frequency | Decision Threshold | |-------|------------|-----------|-------------------| | Development (local testing) | Accuracy, tool calls per task | Every commit | Accuracy ≥ 80%; avg calls ≤ 3 | | Pre-production (staging) | Latency, cost per request, error rate | Daily | Latency < 2s; cost < $0.01/req; error rate < 5% | | Production (live) | Accuracy, latency, cost, user satisfaction | Real-time (Prometheus/CloudWatch) | Accuracy >= 75% (may drop vs. dev); error rate < 2%; Slack alerts if cost > budget | | Post-incident (retrospective) | Which metric failed first? Why? | After each outage | Root-cause analysis; add new assertion |

Each phase has different priorities. Development focuses on accuracy. Production focuses on cost and reliability. Post-incident reviews reveal gaps in monitoring.

Defining success metrics

Correctness

Metric: Accuracy — Fraction of test cases where the agent's answer is correct.

def evaluate_accuracy(agent, test_cases: list[dict]) -> float:
    """
    Run the agent on test cases and measure accuracy.
    Each test case has {"input": str, "expected_output": str}
    """
    correct = 0
    for case in test_cases:
        result = agent.run(case["input"])
        # Simple exact-match comparison (you might use fuzzy matching or LLM-based evaluation)
        if result.strip().lower() == case["expected_output"].strip().lower():
            correct += 1
        else:
            logger.info(f"Failed: {case['input']}\n  Expected: {case['expected_output']}\n  Got: {result}")

    accuracy = correct / len(test_cases) if test_cases else 0
    return accuracy

test_cases = [
    {
        "input": "What is 2 + 2?",
        "expected_output": "4"
    },
    {
        "input": "Who is the current president of France?",
        "expected_output": "Emmanuel Macron"
    },
]

score = evaluate_accuracy(agent, test_cases)
print(f"Accuracy: {score:.1%}")  # e.g., "Accuracy: 85.0%"

For more nuanced evaluation, use an LLM to judge whether answers are equivalent:

def evaluate_with_llm_judge(agent, test_cases: list[dict], judge_model: str = "claude-3-5-sonnet-20241022") -> float:
    """
    Use an LLM to evaluate whether the agent's answer is acceptable.
    More flexible than exact-match for subjective questions.
    """
    correct = 0

    for case in test_cases:
        agent_result = agent.run(case["input"])

        # Ask a judge LLM to evaluate
        judge_prompt = f"""
The user asked: "{case['input']}"
The expected answer was: "{case['expected_output']}"
The agent answered: "{agent_result}"

Is the agent's answer correct or acceptable? Answer YES or NO.
"""
        judge_response = model.messages.create(
            model=judge_model,
            max_tokens=10,
            messages=[{"role": "user", "content": judge_prompt}]
        )

        is_correct = "YES" in judge_response.content[0].text.upper()
        if is_correct:
            correct += 1

    return correct / len(test_cases) if test_cases else 0

Efficiency

Metric: Tool calls per task — How many API/tool calls does the agent make on average?

def evaluate_efficiency(agent, test_cases: list[dict]) -> dict:
    """
    Measure how efficiently the agent uses tools.
    Returns stats like average calls per case, total tokens, latency.
    """
    stats = {
        "total_calls": 0,
        "total_cases": len(test_cases),
        "cases": []
    }

    for case in test_cases:
        # Instrument the agent to track tool calls
        call_count = 0
        original_dispatch = agent.dispatch_tool_call

        def counting_dispatch(call):
            nonlocal call_count
            call_count += 1
            return original_dispatch(call)

        agent.dispatch_tool_call = counting_dispatch

        result = agent.run(case["input"])

        stats["total_calls"] += call_count
        stats["cases"].append({
            "input": case["input"],
            "tool_calls": call_count,
            "result_length": len(result)
        })

        agent.dispatch_tool_call = original_dispatch

    stats["avg_calls"] = stats["total_calls"] / stats["total_cases"] if stats["total_cases"] else 0
    return stats

efficiency = evaluate_efficiency(agent, test_cases)
print(f"Average tool calls per task: {efficiency['avg_calls']:.1f}")
print(f"Total API calls: {efficiency['total_calls']}")

Latency and cost

Metric: Time and tokens — How fast and how expensive is the agent?

import time

def evaluate_latency_and_cost(
    agent,
    test_cases: list[dict],
    token_cost_per_1k: float = 0.003
) -> dict:
    """
    Measure execution time and API cost.
    Assumes token_cost_per_1k is the cost per 1000 tokens (adjust for your model).
    """
    stats = {
        "total_time": 0,
        "total_tokens": 0,
        "cases": [],
        "token_cost_per_1k": token_cost_per_1k
    }

    for case in test_cases:
        start_time = time.time()

        result = agent.run(case["input"])

        elapsed = time.time() - start_time

        # Count tokens (approximate: ~4 chars per token)
        tokens_used = len(result) // 4 + 50  # Rough estimate

        stats["total_time"] += elapsed
        stats["total_tokens"] += tokens_used
        stats["cases"].append({
            "input": case["input"],
            "latency_seconds": elapsed,
            "tokens": tokens_used
        })

    stats["avg_latency"] = stats["total_time"] / len(test_cases) if test_cases else 0
    stats["total_cost"] = (stats["total_tokens"] / 1000) * stats["token_cost_per_1k"]

    return stats

cost_stats = evaluate_latency_and_cost(agent, test_cases, token_cost_per_1k=0.003)
print(f"Average latency: {cost_stats['avg_latency']:.2f}s")
print(f"Total cost for {len(test_cases)} cases: ${cost_stats['total_cost']:.2f}")

Building a test harness

Combine multiple metrics into a single test suite:

import json
from datetime import datetime

class AgentTestSuite:
    """Comprehensive test harness for agents."""

    def __init__(self, agent, test_file: str):
        self.agent = agent
        self.test_cases = self._load_tests(test_file)
        self.results = {}

    def _load_tests(self, file_path: str) -> list[dict]:
        """Load test cases from JSON."""
        with open(file_path, 'r') as f:
            return json.load(f)

    def run_all(self) -> dict:
        """Run all tests and return a summary."""
        results = {
            "timestamp": datetime.now().isoformat(),
            "accuracy": evaluate_accuracy(self.agent, self.test_cases),
            "efficiency": evaluate_efficiency(self.agent, self.test_cases),
            "latency_and_cost": evaluate_latency_and_cost(self.agent, self.test_cases),
            "assertions": []
        }

        # Run assertions
        results["assertions"].append({
            "name": "accuracy_above_80%",
            "passed": results["accuracy"] >= 0.80
        })
        results["assertions"].append({
            "name": "avg_tool_calls_below_3",
            "passed": results["efficiency"]["avg_calls"] < 3
        })
        results["assertions"].append({
            "name": "avg_latency_below_5s",
            "passed": results["latency_and_cost"]["avg_latency"] < 5
        })

        # Determine pass/fail
        all_passed = all(a["passed"] for a in results["assertions"])
        results["status"] = "PASS" if all_passed else "FAIL"

        return results

    def print_summary(self, results: dict):
        """Pretty-print test results."""
        print(f"\n{'='*60}")
        print(f"Agent Test Summary — {results['timestamp']}")
        print(f"{'='*60}")
        print(f"Accuracy: {results['accuracy']:.1%}")
        print(f"Avg Tool Calls: {results['efficiency']['avg_calls']:.1f}")
        print(f"Avg Latency: {results['latency_and_cost']['avg_latency']:.2f}s")
        print(f"Estimated Cost: ${results['latency_and_cost']['total_cost']:.2f}")
        print(f"\nAssertions:")
        for assertion in results["assertions"]:
            status = "✓" if assertion["passed"] else "✗"
            print(f"  {status} {assertion['name']}")
        print(f"\nOverall: {results['status']}")
        print(f"{'='*60}\n")

# Usage:
suite = AgentTestSuite(agent, "test_cases.json")
results = suite.run_all()
suite.print_summary(results)

# Assert fails break the pipeline
assert results["status"] == "PASS", "Agent test suite failed"

Comparing two agent designs

When you refactor or try a new approach, you need to compare against the baseline:

def compare_agents(agent_a, agent_b, test_cases: list[dict]) -> dict:
    """
    Run the same tests on two agents and compare results.
    Returns a side-by-side comparison.
    """
    suite_a = AgentTestSuite(agent_a, None)
    suite_a.test_cases = test_cases
    results_a = suite_a.run_all()

    suite_b = AgentTestSuite(agent_b, None)
    suite_b.test_cases = test_cases
    results_b = suite_b.run_all()

    comparison = {
        "agent_a": results_a,
        "agent_b": results_b,
        "deltas": {
            "accuracy": results_b["accuracy"] - results_a["accuracy"],
            "avg_tool_calls": results_b["efficiency"]["avg_calls"] - results_a["efficiency"]["avg_calls"],
            "avg_latency": results_b["latency_and_cost"]["avg_latency"] - results_a["latency_and_cost"]["avg_latency"],
            "cost": results_b["latency_and_cost"]["total_cost"] - results_a["latency_and_cost"]["total_cost"]
        }
    }

    return comparison

def print_comparison(comparison: dict):
    """Pretty-print agent comparison."""
    print("\nAgent Comparison")
    print(f"{'Metric':<20} {'Agent A':<15} {'Agent B':<15} {'Delta':<15}")
    print("-" * 65)

    print(f"{'Accuracy':<20} {comparison['agent_a']['accuracy']:<14.1%} {comparison['agent_b']['accuracy']:<14.1%} {comparison['deltas']['accuracy']:<14.1%}")
    print(f"{'Avg Tool Calls':<20} {comparison['agent_a']['efficiency']['avg_calls']:<14.1f} {comparison['agent_b']['efficiency']['avg_calls']:<14.1f} {comparison['deltas']['avg_tool_calls']:<14.1f}")
    print(f"{'Avg Latency (s)':<20} {comparison['agent_a']['latency_and_cost']['avg_latency']:<14.2f} {comparison['agent_b']['latency_and_cost']['avg_latency']:<14.2f} {comparison['deltas']['avg_latency']:<14.2f}")
    print(f"{'Total Cost ($)':<20} {comparison['agent_a']['latency_and_cost']['total_cost']:<14.2f} {comparison['agent_b']['latency_and_cost']['total_cost']:<14.2f} {comparison['deltas']['cost']:<14.2f}")

    # Determine winner
    if abs(comparison['deltas']['accuracy']) < 0.02 and comparison['deltas']['cost'] < 0:
        winner = "Agent B (similar accuracy, lower cost)"
    elif comparison['deltas']['accuracy'] > 0.05:
        winner = "Agent B (better accuracy)"
    else:
        winner = "Agent A (baseline or no clear winner)"

    print(f"\nRecommendation: {winner}")

comparison = compare_agents(agent_old, agent_new, test_cases)
print_comparison(comparison)

Case Study: Metrics-Driven Agent Iteration

A financial services company built an agent to classify customer support tickets. They started with a simple accuracy metric: "Does the classification match the ground-truth label?" Initial accuracy was 92%. They deployed it to production feeling confident.

Within a week, they noticed:

• Accuracy was still 92% (measured on new tickets).
• But cost had tripled — the agent was making far more tool calls than before.
• And latency had increased — support reps complained of slow responses.

Root cause: The agent's behavior had shifted. It was now calling the knowledge base tool even for obvious cases, incurring extra cost. A single metric (accuracy) masked two other regressions. The lesson: measure latency and cost alongside accuracy.

After adding cost and latency tracking, they discovered the shift happened because they'd updated the system prompt to be more "cautious." The new version tried to double-check its reasoning before deciding, costing money. They reverted to the old system prompt, kept the same 92% accuracy, but cut cost in half and latency by 40%.

Regression detection

As you iterate, track metrics over time to catch regressions:

import csv

def log_test_result(results: dict, log_file: str = "test_results.csv"):
    """Log results to a CSV for historical tracking."""
    fieldnames = ["timestamp", "accuracy", "avg_calls", "avg_latency", "cost"]

    with open(log_file, 'a', newline='') as f:
        writer = csv.DictWriter(f, fieldnames=fieldnames)
        writer.writerow({
            "timestamp": results["timestamp"],
            "accuracy": results["accuracy"],
            "avg_calls": results["efficiency"]["avg_calls"],
            "avg_latency": results["latency_and_cost"]["avg_latency"],
            "cost": results["latency_and_cost"]["total_cost"]
        })

def detect_regression(log_file: str, threshold_accuracy: float = 0.05) -> bool:
    """
    Check if latest results are a regression from the previous run.
    Returns True if regression detected.
    """
    with open(log_file, 'r') as f:
        lines = list(csv.DictReader(f))

    if len(lines) < 2:
        return False  # Not enough data

    latest = float(lines[-1]["accuracy"])
    previous = float(lines[-2]["accuracy"])

    if latest < previous - threshold_accuracy:
        logger.error(f"REGRESSION: Accuracy dropped from {previous:.1%} to {latest:.1%}")
        return True

    return False

# In your CI/CD pipeline:
suite = AgentTestSuite(agent, "test_cases.json")
results = suite.run_all()
suite.print_summary(results)

log_test_result(results)

if detect_regression("test_results.csv"):
    raise RuntimeError("Test regression detected. Revert changes or investigate.")

Edge case: False confidence from small test sets

A subtle trap occurs when your test set is too small or unrepresentative. Consider:

# Problematic scenario:
# Test set: 10 examples, all similar.
# Agent accuracy: 100% on test set.
# Deployed to production with 100,000 users.
# Accuracy in production: 72%.

# What happened?
# The test set didn't capture edge cases, ambiguous inputs, or user phrasing variations.

To mitigate:

1. Test set size rule: At least 100 examples per major category. For agents, aim for 200–500 to capture variance.
2. Stratification: Ensure your test set has proportional representation of easy, medium, and hard cases.
3. User data: Once in production, continuously audit a sample of user interactions. Did the agent behave as expected? Build a live test set from real usage.
4. Confidence intervals: Don't report accuracy as a point estimate ("92%"). Report it with confidence bounds: "92% ± 3% (95% CI on 200-example test set)."

Common mistake

Over-relying on accuracy as the only metric. A 95% accurate agent is useless if it costs $10 per query. Conversely, a fast, cheap agent that's only 60% accurate wastes user time. Always measure multiple dimensions: accuracy, cost, latency, tool efficiency. The "best" agent is usually a trade-off across all of them, tailored to your use case's priorities.