Context Windows and Their Limits

What a context window is (and isn't)

A context window is the maximum number of tokens a model can accept in a single request. If a model has a 100,000-token context window, that means the sum of all tokens in the system prompt, previous conversation, tool calls, results, and the current user input cannot exceed 100,000. Once you hit that limit, something has to give: either you get an error, the model refuses to respond, or the framework silently truncates older content.

This is different from tokens per se: tokens measure text length; a context window measures a model's capacity. The same 5,000-token piece of text fits comfortably in a 100,000-token window but fills half of a 10,000-token window. Agents care about context windows because tool results, conversation history, and retrieved documents all accumulate within the same window, and a long agent loop can fill it surprisingly fast.

Context window sizes across models (illustrative current landscape)

The following table shows approximate context window sizes as of 2026. Note: these are illustrative estimates; check official documentation for current specifications, as context windows are actively expanding.

| Model | Context window | Output limit | Cost impact notes | |-------|---------|---------|---------| | Claude 3 Haiku | 200k tokens | 4k tokens | Lower cost per token; smaller window feels less constraining at lower absolute cost | | Claude 3.5 Sonnet | 200k tokens | 4k tokens | Medium cost; 200k is enough for most agent loops without aggressive truncation | | Claude 3 Opus | 200k tokens | 4k tokens | Higher per-token cost; large window is valuable for complex reasoning | | GPT-4 Turbo | 128k tokens | 4k tokens | Medium cost; smaller than Claude variants, can hit limits faster | | GPT-4o | 128k tokens | 4k tokens | Lower cost than Turbo; same window size constraints | | Gemini 2.0 | 1M tokens (claimed) | 150k tokens | Still evaluating real-world stability at extreme scales |

The practical insight: a 200k window is enough for most agent loops without aggressive optimization. A 1M window (if stable) enables radically different agent designs with less context churn, but costs are higher if charged per token.

How context fills up in an agent loop

Let's trace what happens turn-by-turn in a real agent:

On turn 1, context is minimal: the system prompt (1,500 tokens), the user's initial question (50 tokens), the model's reasoning (200 tokens), a tool call (50 tokens), and the tool result (300 tokens). Total: about 2,100 tokens. The context window is 10% full.

On turn 5, context is larger: the system prompt (still 1,500), the accumulated conversation history from turns 1-4 (4 × 200 = 800), the four previous tool results (4 × 300 = 1,200), and the current request and tool call (300 tokens). Total: about 3,800 tokens.

On turn 20, if nothing was cleaned up: the system prompt (1,500), all 19 previous turns of conversation and results (19 × 500 = 9,500), and the current request (50). Total: about 11,050 tokens -- the window is at capacity.

At turn 25, the framework has to decide: truncate the oldest turns, summarize them, drop the least-relevant tool results, or refuse the request. Most agent frameworks truncate automatically, but that truncation can remove context that mattered to the current decision. The model's turn-25 reasoning happens without knowing what happened in turns 1-5, which can cause repeated questions, forgotten constraints, or lost progress.

Three practical strategies to manage context over long runs

Strategy 1: Summarization

Rather than keeping every tool result verbatim, periodically compress groups of results into a short summary that preserves the decision-relevant facts and drops the rest. For a research agent, instead of keeping the full text of five web pages (4,000+ tokens), extract the key facts from each, combine them, and summarize: "Found three vendors: Vendor A (good price, 2-week lead time), Vendor B (premium quality, 4-week lead time), Vendor C (budget option, no warranty)." This captures what matters in 100 tokens instead of 4,000.

Here's a practical example using Claude as a summarization tool within an agent:

def summarize_tool_results(results: list[str], summary_budget: int = 200) -> str:
    """
    Compress multiple tool results into a decision-relevant summary
    that fits within a token budget.
    """
    client = anthropic.Anthropic()

    combined = "\n\n".join(results)

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=summary_budget,
        messages=[{
            "role": "user",
            "content": f"""Summarize these tool results in {summary_budget} tokens.
Keep only facts relevant to decision-making. Drop formatting, navigation,
timestamps, and irrelevant details.

Results:
{combined}"""
        }]
    )

    return response.content[0].text

# In your agent loop:
# After every 5 tool calls, compress them into a summary
if step % 5 == 0:
    old_results = context_history[-5:]  # Last 5 results
    summary = summarize_tool_results(old_results)
    context_history = context_history[:-5] + [summary]

This trade-off is explicit: you lose the verbatim tool output but gain the ability to keep running without truncation. For most agent tasks, the summary has enough information to proceed; for rare edge cases where you need the original text, the agent can re-fetch it.

Strategy 2: Sliding window retrieval

Instead of retrieving one large batch of context at the start and carrying it through all remaining steps, retrieve fresh, narrowly-targeted context for what the agent needs right now. This is especially powerful in RAG scenarios.

def retrieve_context_for_step(query: str, vector_store, token_budget: int = 1000) -> str:
    """
    Retrieve only the most relevant documents for the current step,
    sized to fit within a token budget.
    """
    # Embed and search
    results = vector_store.search(query, top_k=10)

    # Accumulate results until budget is exhausted
    context = []
    token_count = 0

    for result in results:
        result_tokens = count_tokens(result.text)
        if token_count + result_tokens > token_budget:
            break  # Stop, don't overflow
        context.append(result.text)
        token_count += result_tokens

    return "\n\n".join(context)

# In your agent loop:
# On each step, retrieve context fresh for that step's goal
current_step_context = retrieve_context_for_step(
    query=current_goal,
    vector_store=docs,
    token_budget=1500
)

This way, turn 20 of your agent loop has context specifically about what turn 20 needs to do, not a stale accumulation of everything from turns 1-19. The tradeoff: it costs time (one retrieval per step instead of one big retrieval upfront) but saves tokens and keeps context focused.

Strategy 3: Compact tool output defaults

Design tools to return a short, structured summary by default, with full detail available on a follow-up call only if the agent actually needs it. This prevents one verbose tool from crowding out everything else.

class ToolResult:
    """A tool result with a compact default view and full-detail option."""

    def __init__(self, summary: str, full_text: str):
        self.summary = summary
        self.full_text = full_text

    def __str__(self):
        # When used as a string in context, return the compact version
        return self.summary

def web_search(query: str) -> ToolResult:
    """
    Search the web, but return a compact result by default.
    """
    results = search_api(query)

    # Compact summary: just title and snippet
    summary = "\n".join([
        f"- {r.title}: {r.snippet}"
        for r in results[:5]
    ])

    # Full text: includes full page content if agent wants it
    full_text = "\n\n".join([
        f"Title: {r.title}\nURL: {r.url}\nContent: {r.full_content}"
        for r in results[:5]
    ])

    return ToolResult(summary, full_text)

# In agent context:
# The agent sees the compact summary (150 tokens)
result = web_search("python async patterns")
print(result)  # Shows summary

# If the agent needs full detail, it calls a separate tool
def get_full_search_result(index: int):
    """Retrieve the full text of a previous search result."""
    # (implementation omitted)
    pass

The agent sees search results as short snippets by default (~150 tokens total), but can explicitly request full detail if it needs to. This keeps context lean for the happy path while preserving the ability to dig deeper if necessary.

A worked example: research agent without and with context management

Without context management:

An agent researches three competing products across 15 steps. Each step calls a search tool, each search returns a full webpage dump (5,000 tokens). By step 10, context is 50,000 tokens of mostly stale webpage text. By step 15, the framework truncates turns 1-5 to stay within the window. The agent on step 15 has forgotten the original requirements from step 1 and repeats searches it already completed.

Tokens per step: ~5,000. Total tokens after 15 steps: ~75,000 (and truncation happens).

With context management:

The same agent, but: search tools return compact summaries (300 tokens); every five steps, older results are summarized into bullet points (100 tokens); and on each step, the agent retrieves a fresh summary of "what have we learned so far" (500 tokens) rather than carrying the full history.

Tokens per step: ~1,000 (300 search + 200 reasoning + 100 summary refresh). Total tokens after 15 steps: ~15,000 (no truncation, and the full context history remains accessible).

The second agent answers the same question 5x more efficiently and with better decision-making because the model can see the requirements alongside the current step.

Understanding truncation policies

Different frameworks and models handle overflow differently:

• Hard limit: the model refuses requests that exceed the window. You get an error, and the agent stops.
• Oldest-first truncation: the framework silently drops the oldest messages/results first, keeping recent context. This is common in chat interfaces.
• Keyword-based truncation: some systems try to keep "important-looking" content (e.g., summaries, constraints) and drop "less important" content (e.g., earlier conversation history).
• Explicit budget: you specify how many tokens each component can use (system prompt: 1,500, history: 2,000, retrieval: 3,000, etc.), and the framework respects those budgets.

The best agent frameworks let you choose or control this. If you don't know how your framework truncates, assume oldest-first (drop early turns first), and design accordingly.

Real-world case study: a customer support agent's context crisis

A customer support platform deployed an agent to help resolve multi-step issues (password reset, billing correction, technical troubleshooting). The agent had a 100k context window, which seemed plenty.

In practice:

• System prompt + tool definitions: 2,000 tokens
• Customer background (history, account type, prior issues): 3,000 tokens
• Current conversation: ~500 tokens per turn
• Each tool result (API call, database lookup): ~1,000 tokens

On a typical issue with 40 turns of back-and-forth, the agent reached context limit by turn 25 and the framework truncated early context (turns 1-5). Turn 25 was left without context about the original problem statement, which was buried in turn 1. The agent got confused and started asking the customer questions it had already asked.

The fix: explicit context budgeting with priority levels. System prompt and customer background were protected (marked non-truncatable). Intermediate results were summarized every 5 turns. Recent context (last 5 turns) was always kept. This gave the model the "skeleton" of the issue (background) plus full recent context (last steps), rather than a disjointed history.

Why raw context window size is a red herring

A model with a 100,000-token window is not 10x better than a model with a 10,000-token window at every task. Research from Anthropic on long-context models consistently shows that:

• Irrelevant context degrades performance, even well within the window. Adding a 50,000-token irrelevant document doesn't help you answer a question; it hurts.
• Attention to detail degradation: models spend less "attention weight" on middle content and more on the beginning and end of context, a phenomenon called "lost in the middle."
• Token efficiency matters more than window size, because you pay for every token, and you pay more per token in frameworks that use longer context.

So a 10,000-token window with ruthlessly curated context often outperforms a 100,000-token window with sloppy context management.

Advanced context strategy: priority-based budgeting

For mission-critical agents, a production-grade approach is explicit priority budgets:

class ContextBudget:
    """Allocate context tokens with priority levels and guarantees."""

    def __init__(self, total_tokens: int):
        self.total = total_tokens
        self.allocated = {}

    def allocate(self, component: str, tokens: int, priority: str = "normal"):
        """
        priority: "critical" (always kept), "normal" (kept if space), "ephemeral" (dropped first)
        """
        self.allocated[component] = {
            "tokens": tokens,
            "priority": priority,
            "actual_used": 0
        }

    def is_over_budget(self) -> bool:
        total_used = sum(c["actual_used"] for c in self.allocated.values())
        return total_used > self.total

    def truncate(self):
        """Intelligently drop content starting with lowest priority."""
        total_used = sum(c["actual_used"] for c in self.allocated.values())
        if total_used <= self.total:
            return  # No truncation needed

        # Drop in order: ephemeral → normal → critical (never)
        for priority in ["ephemeral", "normal"]:
            for comp_name, comp_data in self.allocated.items():
                if comp_data["priority"] != priority:
                    continue
                # Drop this entire component or truncate it
                freed = comp_data["actual_used"]
                comp_data["actual_used"] = 0
                total_used -= freed
                if total_used <= self.total:
                    return

This ensures system prompts and critical constraints are never lost, while ephemeral content (intermediate results, older conversation) is dropped first.

Edge case: context contamination and repeated queries

A subtle failure mode: if an agent loop is poorly designed, the growing context can contain contradictory information. For example:

Turn 1: Agent asks "What is the top vendor?" Tool returns "Vendor A" Turns 2-10: Agent makes decisions based on Vendor A Turn 11: Agent asks the same question again (forgot it already asked); tool returns "Vendor B" (data changed) Turns 12-20: Agent has now seen both answers in context and may get confused about which is current

A production agent should:

• Track tool results in a separate cache (not in the context window alone), so you can distinguish "stale context" from "current information"
• Dedup repeated queries before they bloat context (if the agent calls the same tool twice, reuse the result)
• Explicitly mark "current best information" vs. "historical context" in the prompt

This prevents the model from seeing contradictory tool results within the same context window.

Implementing a deduplication cache for agents

Here's a pattern for preventing duplicate tool calls and context bloat:

class ToolResultCache:
    """
    Cache tool results within an agent run to prevent redundant calls.
    Distinguishes between "current" and "historical" results.
    """

    def __init__(self):
        self.cache = {}  # query_hash -> result
        self.call_order = []  # Track order for prioritization

    def get_or_call(self, tool_name: str, args: dict, tool_func):
        """
        Check cache first. If miss, call tool and cache result.
        """
        # Create a hash of the tool call
        cache_key = (tool_name, tuple(sorted(args.items())))

        if cache_key in self.cache:
            # Reuse cached result
            cached_result = self.cache[cache_key]
            return {
                "result": cached_result,
                "source": "cached",
                "call_number": len(self.call_order)  # When this call was made
            }

        # Cache miss; call tool
        result = tool_func(**args)
        self.cache[cache_key] = result
        self.call_order.append(cache_key)

        return {
            "result": result,
            "source": "fresh",
            "call_number": len(self.call_order)
        }

    def get_summary_for_context(self):
        """
        Return a summary of cached results for the agent context.
        Group by tool and mark which are current vs. historical.
        """
        summary = {}
        for idx, cache_key in enumerate(self.call_order):
            tool_name, args = cache_key[0], dict(cache_key[1])
            result = self.cache[cache_key]

            # Later calls to the same tool supersede earlier ones
            is_current = idx == max(
                i for i, k in enumerate(self.call_order)
                if k[0] == tool_name
            )

            if tool_name not in summary:
                summary[tool_name] = []

            summary[tool_name].append({
                "args": args,
                "result": result,
                "current": is_current,
                "call_order": idx
            })

        return summary

With this cache, when the agent loop inserts context into the prompt, it can explicitly mark "Current information:" (recent results) vs. "Historical context:" (older results that may be outdated). This helps the model disambiguate when the same tool was called multiple times.

Common mistake

Assuming a bigger context window solves context problems on its own. It doesn't. A larger window means the model can hold more tokens, not that everything within it gets equal attention. Agents scale better when you design them to need less context per step, not by relying on a larger window to hold everything. Summarize old results, retrieve fresh context for current steps, and keep tool outputs compact. A 10,000-token window with these techniques outperforms a 100,000-token window without them. And watch out for context contamination: contradictory information in the context (from repeated queries or changing data) confuses models more than omitted information.