Choosing a Model: Cost, Latency, and Quality Tradeoffs

The three-axis model tradeoff

When you choose an LLM for your RAG system, you're optimizing across three dimensions: cost (how much it costs per token), latency (how long it takes to respond), and quality (how well it reasons and avoids hallucinations). No single model wins on all three.

The fundamental tradeoff: Larger, better models cost more and are slower. Smaller, cheaper models are fast but make more mistakes. Your job as a builder is not to find "the best model," but to pick the right model for each task.

Model tiers and their characteristics

Modern LLM providers (Anthropic, OpenAI, Google) offer models in three rough tiers:

Tier 1: Small / Fast Models

Examples: Claude 3.5 Haiku, GPT-4o Mini, Gemini 1.5 Flash

Characteristics:

• Input cost: ~$0.80-2.00 per million tokens (cheapest)
• Output cost: ~$4-10 per million tokens
• Latency: 100-500ms for most queries
• Quality: Good but not excellent; struggles with nuance, multi-step reasoning, or ambiguous judgment calls

Best for:

• Classification & extraction: "Is this support ticket urgent? Classify as: high/medium/low"
• Simple retrievals: "Find the customer's account status"
• Bulk processing: When processing hundreds or thousands of queries, cost matters more than perfection
• High-volume filtering: "Of these 100 search results, which 10 are actually relevant?"
• First-pass summarization: "Give me a one-sentence summary of this chat log"

Real-world cost: A high-volume customer service system using Haiku might process 100,000 queries/day at $0.01-0.02 per query = $1,000-2,000/day total. Switching to a larger model could cost 5-10x more.

Tier 2: Mid-range Models

Examples: Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro

Characteristics:

• Input cost: ~$3-10 per million tokens
• Output cost: ~$12-30 per million tokens
• Latency: 500-2000ms for most queries
• Quality: Excellent; handles nuance, reasoning, and most real-world tasks very well

Best for:

• Most RAG answers: The default choice for question-answering against a knowledge base
• Summarization: Multi-document summaries, meeting notes, long conversations
• Complex extraction: "Extract the product name, price, and three unique selling points from this description"
• Judgment calls: Tasks where some reasoning or nuance is needed
• General-purpose chatbots: As long as you can tolerate 1-2s latency

Real-world cost: A 100K queries/day system using Sonnet might cost $10,000-15,000/day if retrieving 5 chunks of 1000 tokens each (5K input tokens per query). This is expensive, which is why most systems use routing (see below) to avoid paying Sonnet prices for every query.

Tier 3: Frontier / Reasoning Models

Examples: Claude 3 Opus, GPT-4, Gemini 2.0

Characteristics:

• Input cost: ~$15-30 per million tokens (most expensive)
• Output cost: ~$60-100 per million tokens
• Latency: 2-5+ seconds
• Quality: Best-in-class; handles very complex reasoning, ambiguous edge cases, and high-stakes judgment

Best for:

• Final answer quality checks: Before returning an answer to a customer, pass it through Opus to verify it's grounded
• Complex multi-step reasoning: "Analyze this contract for risks, summarize the key terms, and flag anything unusual"
• Rare edge cases: Tasks where the cost of a wrong answer is very high
• Limited-volume, high-stakes queries: Legal review, medical information, policy decisions
• Never for high-volume, routine work; the cost is prohibitive

Real-world cost: Using Opus for every query is unaffordable (costs 10-50x more than Sonnet per query). Used selectively (maybe 5% of queries need Opus-level reasoning), it's a worthwhile insurance policy.

Routing: send different tasks to different models

Most production RAG systems don't use a single model. They use routing logic to decide which model to call based on the query or context.

Simple routing examples:

1. Complexity-based routing:

   • If query is simple (classification, yes/no, direct lookup) → use Haiku
   • If query is moderate (normal RAG, summarization) → use Sonnet
   • If query is complex or high-stakes → use Opus

2. Frequency-based routing:

   • 90% of queries go to Sonnet (the balanced choice)
   • 5% of queries are high-volume/simple and go to Haiku (cheap)
   • 5% of queries need final review and go to Opus (expensive)

3. Confidence-based routing:

   • Sonnet generates an answer
   • If confidence score < 0.6, escalate to Opus for review
   • If confidence score > 0.8, return Sonnet's answer

Here's a routing implementation:

import anthropic

def classify_query_complexity(query: str) -> str:
    """Decide if a query is simple, moderate, or complex."""

    # Simple heuristic: short queries with keywords are often simple
    simple_keywords = ["status", "hours", "price", "location", "contact"]
    complex_keywords = ["why", "compare", "analyze", "recommend", "impact"]

    query_lower = query.lower()

    # Count complexity signals
    complexity_score = 0
    if any(kw in query_lower for kw in complex_keywords):
        complexity_score += 2
    if "?" not in query:  # Statements without questions are often complex requests
        complexity_score += 1
    if len(query) > 200:  # Long queries tend to be more complex
        complexity_score += 1

    # Count simplicity signals
    simplicity_score = 0
    if any(kw in query_lower for kw in simple_keywords):
        simplicity_score += 2
    if query.count("?") == 1:  # Single, direct question
        simplicity_score += 1

    if simplicity_score > complexity_score:
        return "simple"
    elif complexity_score > simplicity_score:
        return "complex"
    else:
        return "moderate"

def rag_answer_with_routing(user_query: str, retrieved_chunks: list[str]) -> dict:
    """
    Answer a RAG query using the right model based on complexity.
    """

    client = anthropic.Anthropic()

    # Determine complexity
    complexity = classify_query_complexity(user_query)

    # Choose model based on complexity
    if complexity == "simple":
        model = "claude-3-5-haiku-20241022"  # Cheapest
        max_tokens = 300  # Simple answers are short
    elif complexity == "moderate":
        model = "claude-3-5-sonnet-20241022"  # Best balance
        max_tokens = 500
    else:  # complex
        model = "claude-3-opus-20250219"  # Best quality
        max_tokens = 1000

    # Format context
    context = "Retrieved context:\n"
    for i, chunk in enumerate(retrieved_chunks):
        context += f"[Source {i}]\n{chunk}\n"

    prompt = f"""{context}

User question: {user_query}

Answer the question using ONLY the provided context. If the context doesn't contain the answer, say so."""

    response = client.messages.create(
        model=model,
        max_tokens=max_tokens,
        messages=[{"role": "user", "content": prompt}]
    )

    answer = response.content[0].text

    # Count input tokens for cost calculation
    input_tokens = response.usage.input_tokens
    output_tokens = response.usage.output_tokens

    # Estimate cost (illustrative estimate)
    cost_per_million = {
        "claude-3-5-haiku-20241022": {"input": 0.80, "output": 4.00},
        "claude-3-5-sonnet-20241022": {"input": 3.00, "output": 15.00},
        "claude-3-opus-20250219": {"input": 15.00, "output": 75.00}
    }

    rates = cost_per_million.get(model, {"input": 3.00, "output": 15.00})
    cost = (input_tokens * rates["input"] + output_tokens * rates["output"]) / 1_000_000

    return {
        "answer": answer,
        "model_used": model,
        "complexity": complexity,
        "cost_estimate": cost,
        "tokens": {"input": input_tokens, "output": output_tokens}
    }

# Usage examples
print(rag_answer_with_routing("What's your phone number?", ["Contact: 555-1234"]))
# → Uses Haiku (simple), costs ~$0.001

print(rag_answer_with_routing(
    "Why is our product better than competitors? Consider features, price, and customer reviews.",
    ["Our product has X, Y, Z features...", "Competitors: A has...", "Customer reviews: ..."]
))
# → Uses Sonnet (complex), costs ~$0.02

Combining multiple models in a pipeline

For maximum reliability and cost-effectiveness, some systems use a pipeline:

1. Tier 1 (Haiku): Initial answer generation from retrieved chunks
2. Tier 2 (Sonnet): Confidence check—is the answer grounded in retrieved context?
3. Tier 3 (Opus): Final review for high-stakes queries (legal, medical, financial advice)

def tiered_rag_pipeline(query: str, chunks: list[str]) -> dict:
    """
    Multi-tier pipeline: Haiku → Sonnet (confidence check) → Opus (final review).
    """

    client = anthropic.Anthropic()

    context = "Retrieved context:\n" + "\n".join(chunks)
    prompt_template = f"""{context}

Question: {{question}}

{{instruction}}"""

    # Stage 1: Haiku generates initial answer
    stage1_response = client.messages.create(
        model="claude-3-5-haiku-20241022",
        max_tokens=400,
        messages=[{"role": "user", "content": prompt_template.format(
            question=query,
            instruction="Answer using the retrieved context."
        )}]
    )

    initial_answer = stage1_response.content[0].text

    # Stage 2: Sonnet checks confidence
    stage2_response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=300,
        messages=[{"role": "user", "content": f"""Original question: {query}

Initial answer: {initial_answer}

Retrieved context: {context}

Is this answer grounded in the retrieved context? Rate confidence 0-1."""}]
    )

    confidence_text = stage2_response.content[0].text

    # Extract confidence score from response
    try:
        confidence = float([s for s in confidence_text.split() if 0 <= float(s) <= 1][0])
    except:
        confidence = 0.5

    result = {
        "answer": initial_answer,
        "confidence": confidence,
        "stage_used": "haiku+sonnet"
    }

    # Stage 3: If low confidence, escalate to Opus
    if confidence < 0.6:
        stage3_response = client.messages.create(
            model="claude-3-opus-20250219",
            max_tokens=500,
            messages=[{"role": "user", "content": prompt_template.format(
                question=query,
                instruction="This is a high-stakes question. Answer carefully, grounded in the context."
            )}]
        )

        result["answer"] = stage3_response.content[0].text
        result["stage_used"] = "haiku+sonnet→opus_escalation"

    return result

This pipeline keeps costs low (most queries use cheap Haiku) while ensuring high-confidence answers are reviewed by a better model.

Comparison table: model choice by task

Worked example: routing in a customer service system

A customer service RAG chatbot gets 10,000 queries per day. Queries vary from simple (order status) to complex (product comparison, complaint escalation).

Without routing:

• Use Sonnet for all queries
• Cost: 10,000 queries/day × 5K input tokens × $3/1M = $150/day = $4,500/month

With routing:

• 60% of queries are simple → use Haiku at $0.002/query = $1,200/month
• 35% of queries are moderate → use Sonnet at $0.015/query = $1,575/month
• 5% of queries are complex → use Opus at $0.100/query = $150/month
• Total: ~$2,925/month (35% reduction vs. all-Sonnet)

Quality difference: Haiku misclassifies 5% of simple queries (sends easy questions to a human), but those are rare and cheap to fix. Overall system quality improves because Sonnet and Opus get more resources for harder queries.

Common mistake

Assuming bigger is always better. Upgrading from Sonnet to Opus doesn't improve simple queries (Haiku + Sonnet routing is better). And throwing Opus at every query is wasteful; most RAG queries are straightforward and don't need frontier-level reasoning.

Also common: not measuring the actual cost impact. The difference between models matters when running thousands of queries per day. A $0.01 difference per query × 10,000 queries/day = $100/day = $3,000/month. That's real money, so it's worth optimizing.

The right approach: measure your query distribution, classify by complexity, and route accordingly. Most systems end up using Sonnet as the default, Haiku for high-volume simple queries, and Opus selectively for the hardest 5-10% of queries.