Context Windows and Token Economics for Builders

What a context window actually is

A context window is the maximum amount of text (measured in tokens) that a model can see at once. Claude 3.5 Sonnet has a 200K token window. GPT-4 Turbo has 128K. Older models like GPT-3.5 have 4K. This means:

• A single API call to Claude can include up to 200,000 tokens of input (system message, conversation history, retrieved documents, user query)
• Anything beyond that is dropped or rejected
• The model can't "see" outside the window—tokens that don't fit are gone

For RAG specifically, this matters a lot. In a typical RAG call:

• System prompt: 500-1000 tokens (fixed overhead)
• Conversation history: 0-5000 tokens (scales with conversation length)
• Retrieved context (chunks): 2000-15000 tokens (scales with chunk count and size)
• User query: 50-500 tokens
• Model output: 0-2000 tokens (reserved space for the response)

If you have a 4K context window model and you're trying to retrieve 5 chunks of 2000 tokens each, you've already used 10K tokens—more than double the window. The model sees only the first chunk, and retrieval effectively fails.

Longer context windows let you stuff more retrieved chunks into a single call, which usually improves answer quality (more evidence, broader context). But they come with tradeoffs: cost per token is usually higher, latency is slower, and the model sometimes gets confused by "lost in the middle" problems (it focuses on early and late information and forgets the middle).

Where token cost concentrates in RAG

Here's the cost breakdown for a typical RAG request (illustrative estimate, not a vendor benchmark):

| Component | Tokens | Percentage | Cost driver | |---|---|---|---| | System prompt | 200-500 | ~5% | Fixed per call | | Conversation history | 1000-3000 | ~10% | Scales with # of turns | | Retrieved context (chunks) | 8000-12000 | 60-75% | Scales with chunk count × chunk size | | User query | 100-300 | ~2% | Usually small | | Output (response) | 200-1000 | ~5% | Scales with answer verbosity |

The retrieved context is your main cost lever. If you retrieve 10 chunks at 1000 tokens each, you're burning 10K input tokens. If you reduce to 5 chunks of 600 tokens each, you're down to 3K—a 70% cost reduction.

This means RAG cost optimization is almost always about:

1. Retrieve fewer chunks (confidence thresholds on retrieval scores)
2. Use smaller chunks (200-400 tokens instead of 1000+)
3. Use cheaper retrieval models (BM25 keyword search instead of embeddings, which saves the embedding model cost)
4. Cache repeated context (if the same knowledge is needed across many queries, cache it)

System prompt is usually the smallest cost. Even a detailed 1000-token system prompt is dwarfed by 10K tokens of retrieved context. Don't over-optimize the system prompt; optimize retrieval instead.

Output cost is usually cheap. Even a verbose 1000-token response is still a tiny fraction of the total. You're paying for input, not output (in most RAG workflows).

Prompt caching to reduce redundant costs

If you're running many queries against the same knowledge base, you'll often retrieve overlapping chunks or use the same system prompt repeatedly. Prompt caching (supported by Claude and GPT-4) lets you tell the model "I'm about to send you a long piece of text that I'll send again later, so cache it."

The first time you send a cached section, you pay full price. Subsequent requests reuse the cache for 90% cost reduction on that section. Cache hits persist for 5 minutes (in Claude, or 1 hour in GPT-4 Turbo).

Example: A customer support RAG system answers 100 queries per hour about the same 20 support documents. Without caching, you embed and retrieve those same documents 100 times. With caching, you pay full price once, then 90% cheaper for 99 more times.

Here's how to implement cache hits in a RAG system:

import anthropic

client = anthropic.Anthropic(api_key="your-api-key")

# Your knowledge base (same for all queries in this batch)
SYSTEM_CONTEXT = """You are a customer support agent.
Use the provided documents to answer questions.
If the answer isn't in the documents, say so."""

KNOWLEDGE_BASE = """# Company Policies
Return policy: 30 days for unopened items, 14 days if opened.
Shipping: Free over $50, $5 flat otherwise.
Warranty: 1 year parts and labor..."""  # This repeats across queries

def answer_query_with_cache(user_query: str):
    """Answer a query, reusing cached knowledge base."""

    # Claude's cache feature: mark long sections with cache_control
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=500,
        system=[
            {
                "type": "text",
                "text": SYSTEM_CONTEXT
            },
            {
                "type": "text",
                "text": f"Knowledge Base:\n{KNOWLEDGE_BASE}",
                # This tells Claude to cache the knowledge base
                "cache_control": {"type": "ephemeral"}
            }
        ],
        messages=[
            {
                "type": "user",
                "content": user_query
            }
        ]
    )

    # Check cache usage (in usage stats)
    usage = response.usage
    print(f"Input tokens: {usage.input_tokens}")
    print(f"Cache creation tokens: {usage.cache_creation_input_tokens or 0}")
    print(f"Cache read tokens: {usage.cache_read_input_tokens or 0}")

    return response.content[0].text

# First query: pays full price for knowledge base
answer = answer_query_with_cache("What's your return policy?")
# Usage: 5000 input tokens + 8000 cache creation tokens

# Second query (within 5 min): reuses cached knowledge base
answer = answer_query_with_cache("Do you offer free shipping?")
# Usage: 200 input tokens + 8000 cache read tokens (90% cheaper)

The math: if caching reduces cached tokens to 10% cost, and your cached section is 8000 tokens, then per cached query you save 7200 tokens of cost—a massive win if you're running hundreds of queries.

When to use caching:

• Knowledge base is stable (doesn't change per query)
• Queries come in batches (more than 5 in a 5-minute window)
• Same retrieval result is used across multiple queries

When caching doesn't help:

• Every query retrieves different chunks (no repetition)
• Queries are sparse (one every hour—cache expires)

Context window overflow: what happens when you run out?

When you exceed the model's context window, one of two things happens:

1. Truncation: The model receives only the first N tokens that fit, and the rest is dropped. If you retrieved 5 chunks but only 2 fit in the window, the model only sees 2. The RAG system silently degrades.

2. Rejection: The API returns an error: "Input exceeds max tokens for model." Your application crashes or falls back to a degraded mode.

Neither is good. Here's how to handle it defensively:

def rag_with_overflow_handling(query: str, retrieved_chunks: list[str], max_tokens: int = 200000):
    """
    RAG with defensive context window management.
    Ensures we never exceed the model's context limit.
    """

    client = anthropic.Anthropic()

    # Constants for this model
    SYSTEM_PROMPT = "You are a helpful assistant. Answer based on the provided context only."
    RESERVED_FOR_OUTPUT = 2000  # Space for the model's response
    SAFETY_MARGIN = 500  # Buffer to be safe

    # Available budget for input
    available_input_budget = max_tokens - RESERVED_FOR_OUTPUT - SAFETY_MARGIN

    # Calculate token count (approximate: 1 token ≈ 4 characters)
    system_tokens = len(SYSTEM_PROMPT) // 4
    query_tokens = len(query) // 4

    # Budget left for retrieved context
    context_budget = available_input_budget - system_tokens - query_tokens

    # Fit as many chunks as possible within the budget
    context_text = ""
    chunks_included = 0

    for chunk in retrieved_chunks:
        chunk_tokens = len(chunk) // 4

        # If adding this chunk would exceed budget, stop
        if len(context_text) // 4 + chunk_tokens > context_budget:
            break

        context_text += f"\n---\n{chunk}"
        chunks_included += 1

    # If we couldn't fit any chunks, we have a serious problem
    if chunks_included == 0:
        # Fallback: use a single, shorter chunk
        if retrieved_chunks:
            context_text = retrieved_chunks[0][:1000]
            chunks_included = 1

    print(f"Context window: {max_tokens}, Used: ~{len(context_text) // 4}, Chunks: {chunks_included}/{len(retrieved_chunks)}")

    try:
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=min(RESERVED_FOR_OUTPUT, 2000),
            system=SYSTEM_PROMPT,
            messages=[
                {
                    "type": "user",
                    "content": f"Context:\n{context_text}\n\nQuestion: {query}"
                }
            ]
        )

        return {
            "answer": response.content[0].text,
            "chunks_used": chunks_included,
            "total_chunks_available": len(retrieved_chunks),
            "truncated": chunks_included < len(retrieved_chunks)
        }

    except Exception as e:
        # If we still hit a limit error, something is very wrong
        # Log it and return a graceful error
        return {
            "answer": "I encountered an error processing your question. Please try a simpler query.",
            "error": str(e),
            "chunks_used": 0
        }

# Usage
chunks = [
    "Return policy: 30 days unopened...",
    "Shipping costs: Free over $50...",
    # ... more chunks
]

result = rag_with_overflow_handling("What's your return policy?", chunks)
print(result)

Key defensive practices:

1. Count tokens before calling the API (approximate or exact)
2. Reserve output space (don't assume the model can use the entire window for input)
3. Gracefully degrade when you can't fit all context (prefer 3 good chunks over 0 chunks or an error)
4. Monitor and alert when truncation happens (log when chunks are dropped)

Token economics: choosing your model

Larger models are more capable but more expensive. Here's how token cost affects model choice:

| Model | Input cost per 1M | Output cost per 1M | Best for | |---|---|---|---| | Claude 3.5 Haiku | $0.80 | $4.00 | Cost-sensitive, high-volume | | Claude 3.5 Sonnet | $3.00 | $15.00 | Most RAG tasks, good quality/cost | | Claude 3 Opus | $15.00 | $75.00 | Complex reasoning, final answers | | GPT-4 Turbo | $10.00 | $30.00 | Complex tasks, high quality |

For RAG, Haiku or Sonnet are usually the right choice. Opus is overkill unless you're doing multi-step reasoning that requires maximum intelligence. But token economics vary by use case:

• High-volume, simple retrieval: Use Haiku. At $0.80 per million tokens, even 10K tokens per query costs less than $0.01.
• Complex answers: Use Sonnet. The cost difference ($0.002 per query) is worth better quality.
• Final answer or review: Use Opus. You can afford it because you're only calling it once per conversation.

Routing strategy: Many production systems use Haiku for initial retrieval and Sonnet for final answer generation. This is called "cost-aware routing" and it cuts costs 50%+ while keeping quality high.

Worked example: Cost calculation for a support chatbot

A customer support chatbot answers 10,000 queries per day. Each query:

• System prompt: 300 tokens
• Retrieved context: 5 chunks × 800 tokens = 4000 tokens
• Query: 150 tokens
• Response: 300 tokens (average)

Total per query: ~4750 tokens input + 300 tokens output

Daily volume: 10,000 × (4750 input + 300 output) = 50.5M tokens

Cost with Claude 3.5 Sonnet: (50M × $3/1M) + (3M × $15/1M) = $150 + $45 = $195/day

Cost with Claude 3.5 Haiku: (50M × $0.80/1M) + (3M × $4/1M) = $40 + $12 = $52/day

Cost difference: $143/day = $4,290/month. Switching from Sonnet to Haiku for first-pass retrieval saves the company $50K+ per year (Sonnet for final review still available, but only called for 5% of queries that need escalation).

This is why token counting and context management matter: a tiny cost per query × 10,000 queries × 365 days turns into real money.

Common mistake

Optimizing the wrong component. Teams often try to reduce costs by:

• Shortening the system prompt (saves ~100 tokens, ~0.1% of cost)
• Upgrading to a faster model (faster isn't cheaper; often more expensive)
• Caching when there's no cache hit (reduces cost by 0%, but adds latency)

The actual cost leverage is in:

• Chunk size: Reduce from 1000 to 500 tokens per chunk = 50% cost reduction
• Chunk count: Retrieve 3 instead of 10 chunks = 70% cost reduction
• Model selection: Use Haiku instead of Opus = 80% cost reduction
• Prompt caching: Reuse knowledge across queries = 90% cost reduction on cached parts

Focus on retrieved context, not system prompts. That's where the tokens live.