Allocating Your Context Budget

Every API call to a language model has a fixed context window — the maximum number of tokens (roughly words) you can send in and receive back. Claude 3.5 Sonnet allows up to 200,000 tokens. OpenAI's GPT-4o allows 128,000. Smaller models or older versions offer 4,000 to 32,000. When you send system instructions, a document to analyze, prior conversation history, and a new question, you are competing for a finite budget. This lesson teaches you how to allocate that budget strategically.

Context windows are both a blessing and a curse. The large windows (200K tokens) let you include substantial documents, long conversations, and detailed instructions all in one request. But teams often waste this abundance by including everything, resulting in slow, expensive API calls where most of the input is noise. The art is making strategic choices: what must you include? What can you trim? What can you summarize?

Understanding the Context Window

A typical API call consists of:

1. System instructions (~100–500 tokens): "You are an analyst. Be concise. Cite sources."
2. Retrieved context or documents (100–50,000+ tokens): The file, article, or data the user wants you to analyze.
3. Conversation history (0–10,000 tokens): Prior messages in a multi-turn chat.
4. Current user question/request (50–1,000 tokens): "What is the main risk?"

The sum of all four cannot exceed your model's context window. For concreteness, assume you are using Claude 3.5 Sonnet with a 200,000-token window. A typical budget allocation:

| Section | Typical Tokens | % of Budget | Notes | |---------|---|---|---| | System instructions | 300 | 0.15% | Small but critical. Defines task and tone. | | Retrieved documents | 80,000 | 40% | The payload. Usually the largest section. | | Conversation history | 20,000 | 10% | Multi-turn context. Can be compressed. | | Current question | 500 | 0.25% | Minimal. Usually very small. | | Reserved buffer | 99,200 | 49.6% | Safety margin to avoid hitting hard limit. Never use 100% of window. |

Why the buffer? The model's output counts against the window. If you allocate 100% for input, the model cannot generate a response. A buffer ensures the model has room to think and respond.

Sizing Each Section

System Instructions

Aim for 200–500 tokens. A comprehensive system instruction might be:

You are an expert legal analyst with 15 years of corporate law experience.
Your task is to review contracts and identify risks.

Respond in this format:
1. Key risks (3–5 bullets)
2. Recommended clauses (2–3 bullets)
3. Negotiation priorities (1–2 sentences)

Be precise. Cite clause numbers. Flag any ambiguity.

This is roughly 80 tokens. Even a detailed, multi-step system instruction rarely exceeds 500 tokens. Avoid redundancy. If you find your system instruction is over 1,000 tokens, you are probably repeating guidance or including content that belongs in the document section.

Retrieved Context / Documents

This is your primary payload. How much should you allocate?

Strategy 1: Fit the whole document (if small).

If the document is under 10,000 tokens (roughly 3,000–4,000 words), include the whole thing. Small documents fit comfortably.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
doc = "long document text..."
tokens = len(enc.encode(doc))
print(f"Document: {tokens} tokens")

if tokens < 10000:
    # Include whole document
    context = doc
else:
    # Summarize or truncate
    context = summarize_or_chunk(doc)

Strategy 2: Summarize or excerpt (if large).

If the document is 10,000–100,000 tokens, you must choose:

a) Summarize first: Prompt a model to summarize the document, then use the summary.

Summarize this 50-page report in 5 bullet points, under 1,000 tokens total.
Keep facts and figures. Drop fluff.

[Report text...]

Result: 50,000 tokens → 800 tokens.

b) Extract relevant sections: Use keyword search or semantic similarity to find the ~5 most relevant pages.

# Pseudo-code
relevant_sections = search_document(document, user_question, top_k=5)
context = "\n\n".join(relevant_sections)  # ~5,000–10,000 tokens instead of 50,000

c) Chunk and iterate: If the document is enormous and you cannot summarize it, split it into chunks and process iteratively.

Step 1: Analyze pages 1–10. Report findings.
Step 2: Analyze pages 11–20. Add to findings.
...
Step N: Synthesize all findings.

This trades latency (N API calls) for context efficiency.

Strategy 3: Dynamic allocation.

Allocate context based on the question's complexity:

context_budget = 200000  # Total window
reserved_buffer = context_budget * 0.2  # 20% buffer for output
available = context_budget - reserved_buffer  # 160,000 tokens available

system_tokens = 300
question_tokens = len(enc.encode(user_question))
remaining_for_docs = available - system_tokens - question_tokens

# Allocate remaining budget to docs and history
if len(conversation_history_tokens) < remaining_for_docs * 0.2:
    doc_budget = remaining_for_docs * 0.8
else:
    # History is large; reduce doc budget
    history_budget = remaining_for_docs * 0.3
    doc_budget = remaining_for_docs * 0.7

Conversation History

In multi-turn conversations, older messages are less relevant than recent ones. Trim aggressively:

Rule: Keep last N turns, drop earlier turns.

max_history_tokens = 15000
history = []
token_count = 0

# Walk backward through conversation
for message in reversed(conversation_messages):
    msg_tokens = len(enc.encode(message['content']))
    if token_count + msg_tokens > max_history_tokens:
        break
    history.insert(0, message)
    token_count += msg_tokens

# Use 'history' (most recent messages only)

Alternatively: Summarize old history.

These are the previous turns in the conversation [summary of turns 1–10].
Current turn (turn 11): User asks [question].

By summarizing old turns into a single paragraph, you preserve context without the token cost.

What to Trim When Over Budget

If your total input exceeds the context window, trim in this order:

1. Conversation history (easiest to trim; summarize or drop oldest turns).
2. Retrieved documents (excerpts; use semantic search to keep only relevant sections).
3. System instructions (last resort; condense to essentials).
4. Never trim the current user question (defeat the purpose).

Example decision tree:

Total tokens = 250,000 (exceeds 200,000 limit)

Step 1: Is history > 15,000?
  Yes → Trim history to last 5 turns. New total: 220,000.

Step 2: Still over?
  Yes → Extract top 3 relevant document sections. New total: 180,000.

Step 3: Still over or under-resourced?
  No → Proceed.

Worked Example 1: Contract Review Task

Scenario: You have a 45-page contract (60,000 tokens), user's specific question (~100 tokens), and 8 prior messages of context (~5,000 tokens). Your model has a 100,000-token window.

Step-by-step allocation:

Total available: 100,000 tokens
Reserved buffer: 20,000 tokens (20% — never use 100% of limit)
Available for input: 80,000 tokens

System instructions: 300 tokens
User question: 100 tokens
Conversation history: 5,000 tokens

Subtotal (non-document): 5,400 tokens
Remaining budget for document: 80,000 - 5,400 = 74,600 tokens

Can we fit the contract? Contract is 60,000 tokens. Yes, it fits within 74,600. Include the whole contract.

Token accounting code:

import anthropic
import tiktoken

client = anthropic.Anthropic()
enc = tiktoken.encoding_for_model("gpt-4")  # Approximation tool

system_msg = "You are an expert legal analyst..."
contract_text = "[45 pages...]"
user_question = "What are the indemnification clauses?"
history = "[Prior 8 messages...]"

# Count tokens
system_tokens = len(enc.encode(system_msg))  # ~300
contract_tokens = len(enc.encode(contract_text))  # ~60,000
question_tokens = len(enc.encode(user_question))  # ~20
history_tokens = len(enc.encode(history))  # ~5,000

window = 100000
buffer = int(window * 0.2)
available = window - buffer

total = system_tokens + contract_tokens + question_tokens + history_tokens

if total > available:
    print(f"Over budget by {total - available} tokens. Truncate contract.")
    # Implement truncation strategy
else:
    print(f"Fits! Budget headroom: {available - total} tokens")

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=20000,  # Space for response
    messages=[{
        "role": "user",
        "content": f"{system_msg}\n\nPRIOR CONTEXT:\n{history}\n\nCONTRACT:\n{contract_text}\n\nQUESTION:\n{user_question}"
    }]
)

What if the contract was 200,000 tokens?

Available: 74,600 tokens
Contract size: 200,000 tokens
Fit? NO — exceeds by 125,400 tokens

Options (ranked by information preservation):
1. Extract relevant sections (e.g., "liability", "indemnification", "termination")
   → Estimated: 10,000–15,000 tokens
   → Fits? YES. Proceed with this option.

2. Summarize entire contract to 1-2 pages
   → Estimated: 5,000–8,000 tokens
   → Fits? YES. But loses detail.

3. Split contract into 3 chunks, analyze each, synthesize
   → Cost: 3 API calls (3× the latency and cost)
   → Quality: High (every page reviewed)

Choose option 1: Extract relevant sections via keyword search, then send.

Worked Example 2: Multi-Turn Analysis with Progressive Summarization

Scenario: A user analyzes their finances with you over 15 turns. Each turn adds context. By turn 10, history is eating your budget.

Turn-by-turn allocation strategy:

Turn 1: Income $5K/month, expenses $3K. Budget: 500 tokens used. Headroom: 79,500.
Turn 2: User question about taxes. New total: 700 tokens. Headroom: 79,300.
Turn 3: User mentions investments. Total: 900 tokens. Headroom: 79,100.
...
Turn 10: History is now 10,000 tokens (no summarization). Headroom: 70,000.

Problem: History growing linearly. By turn 30, history would be 30,000 tokens.

Solution: Progressive summarization.

Turn 11+: Before adding the new turn, summarize turns 1–10 into 1,000 words (~300 tokens).
New history: [Summary of turns 1–10, 300 tokens] + [Turn 11, 200 tokens] = 500 tokens.
Headroom restored: 79,500 tokens.

This keeps context usage constant (~500 tokens/turn) while preserving key facts.

Implementation:

def summarize_old_turns(turns_1_to_n, target_tokens=300):
    """Summarize old turns to save context budget."""
    summary_prompt = f"""Summarize this financial conversation in {target_tokens // 4} words.
Keep: income, expenses, assets, liabilities, goals. Drop: casual chatter.

Turns:
{chr(10).join([t['content'] for t in turns_1_to_n])}

Summary:"""

    summary = call_model(summary_prompt)
    return summary  # ~300 tokens

# In multi-turn loop:
if len(conversation_history_tokens) > 8000:
    # Summarize old turns
    summary = summarize_old_turns(turns[:-2])  # Summarize all but last 2 turns
    conversation_history = [summary, turns[-1], turns[-2]]  # Keep last 2 turns in detail

Token Counting in Production

Always measure before sending:

import anthropic
import tiktoken

client = anthropic.Anthropic()

def count_tokens(text, model="claude-3-5-sonnet-20241022"):
    # Anthropic counts tokens server-side; you can also use tiktoken as approximation
    return len(tiktoken.encoding_for_model("gpt-4").encode(text))  # Rough approximation

def allocate_context(system_msg, documents, history, question, window_size=200000):
    buffer = int(window_size * 0.2)
    available = window_size - buffer

    system_tokens = count_tokens(system_msg)
    question_tokens = count_tokens(question)
    history_tokens = count_tokens(history)
    doc_tokens = count_tokens(documents)

    total = system_tokens + question_tokens + history_tokens + doc_tokens

    if total > available:
        print(f"Over budget by {total - available} tokens")
        print("Trimming history...")
        # Trim logic here

    return {"system": system_msg, "docs": documents, "history": history, "question": question}

Compression Techniques: Maximizing Signal in Limited Space

When you're over budget, you can compress before trimming:

1. Summarize Before Including

Instead of: "Here's a 100-page manual: [full text]" Do: "Here's a 5-page summary of the manual: [summary]"

Cost: One summarization call (cheap) to save 50K tokens downstream.

2. Extract Key Points (Semantic Compression)

# Instead of full document, extract only relevant sections
relevant_sections = semantic_search(document, user_query, top_k=3)
context = "\n---\n".join(relevant_sections)  # ~5K tokens instead of 50K

This keeps domain-specific information while discarding generic material.

3. Use Structured Summaries for Conversations

Instead of keeping full turn-by-turn history, compress to:

Summary of conversation turns 1–10:
- User goal: [1 sentence]
- Key decisions made: [3–5 bullets]
- Current status: [1 sentence]

This captures intent without token bloat.

Advanced: Measuring and Tracking Context Spend

In production, you need visibility into actual token usage per request. Here's a monitoring pattern:

import anthropic

client = anthropic.Anthropic()

def call_with_tracking(system_msg, user_msg, model="claude-3-5-sonnet-20241022"):
    response = client.messages.create(
        model=model,
        max_tokens=2000,
        system=system_msg,
        messages=[{"role": "user", "content": user_msg}]
    )

    usage = response.usage
    print(f"Input: {usage.input_tokens}, Output: {usage.output_tokens}, Total: {usage.input_tokens + usage.output_tokens}")

    # Log if over threshold
    if usage.input_tokens > 100000:
        log_high_context_alert(model, usage.input_tokens)

    return response

# In production, aggregate these metrics to find which types of requests use most context

By tracking usage, you can identify which request patterns are context-heavy and optimize them.

Budget Allocation Decision Framework

| Situation | Total Needed | Available | Strategy | |-----------|---|---|---| | Document + history fits easily | 30,000 | 80,000 | Include both fully. No tradeoff. | | Document tight, history large | 75,000 | 80,000 | Keep full document, compress history (summarize or drop old turns) | | Document huge, history small | 85,000 | 80,000 | Extract relevant sections from document, keep history | | Both large | 150,000 | 80,000 | Extract doc sections + summarize history. Consider chunking. | | Ambiguous fit (60K total, 80K budget) | 60,000 | 80,000 | Safe. Add 20K buffer. Proceed. | | Exactly at limit (75K, 75K budget) | 75,000 | 75,000 | DANGER. No room for response. Reduce to 60K input. |

Golden rule: Never use more than 80% of window for input. Always reserve 20% for model output.

Common Mistake

The mistake: Including entire multi-thousand-page documents without considering the context budget.

I need you to analyze this 500-page financial report.
[Entire report pasted in, 750,000 tokens]
What is the risk?

Result: Over budget. API returns an error or silently truncates. The model never sees the full document. Or, if the provider has generous limits, you pay 3–5× more than necessary for a slower response.

The fix: Measure first, allocate strategically.

report_tokens = count_tokens(full_report)  # 750,000

if report_tokens > available_budget:
    # Option 1: Summarize (fast, loses detail)
    summary = summarize_report(full_report, target_tokens=20000)

    # Option 2: Extract chapters (balanced)
    relevant_chapters = extract_chapters_by_keyword(full_report, ["risk", "liability", "reserves"])

    # Option 3: Chunk and iterate (thorough, multiple calls)
    chunks = [full_report[i:i+20000] for i in range(0, len(full_report), 20000)]
    analyses = [analyze_chunk(chunk) for chunk in chunks]
    final_synthesis = synthesize_analyses(analyses)

Choose the strategy that fits your budget, use case, and latency requirements. Summarizing saves tokens but loses detail. Chunking preserves detail but costs more. Extract relevant sections is usually the sweet spot.

Real-World Scenario: Multi-Turn Analysis Task

Situation: A user wants to chat with you about their financial situation across multiple turns. Turn 1, they describe their income and expenses (300 tokens). You respond with initial advice (200 tokens). Turn 2, they ask a follow-up about taxes (100 tokens). You should respond with personalized advice that refers back to their original context.

Naive approach (include everything):

Total context = System instructions (300) + Turn 1 full context (500) + Turn 2 (100) + buffer = 900 tokens

Problem: After 10 turns, you have 3,000+ tokens of context where much of it is repetitive or outdated.

Smart approach (progressive summarization):

Turn 1: Keep full context (500 tokens).
Turn 2: Summarize Turn 1 into 100 tokens. Include Turn 1 summary + Turn 2 (200 tokens total, save 300 tokens).
Turn 3: Summarize Turns 1–2 into 150 tokens. Include summary + Turn 3 (250 tokens total, save 500 tokens).
...
Turn 10: Include only the last 2 turns in detail, earlier turns as a 100-word summary.

This keeps context usage roughly constant (~500 tokens per turn) while preserving all essential information.

Context budget allocation is not glamorous, but it is fundamental to building reliable AI systems. Measure your tokens, allocate strategically, trim in order of priority, and always reserve a buffer. When you do this well, every token counts toward your goal instead of being wasted on unnecessary repetition or dropped entirely. And when you ignore context budgets, costs explode, latency increases, and your model's response quality sometimes actually decreases because it is distracted by irrelevant context.