Context Engineering for Agents

Prompt engineering vs. context engineering

Prompt engineering is about wording a single message well. "How should I phrase the system prompt to get the best results?"

Context engineering is the broader discipline of deciding, on every single turn of a long-running agent, what earns a place in a limited context window out of everything that could go in it. On turn 1, your context is mostly system prompt. On turn 20, it's:

• System prompt (still there)
• Conversation history (19 turns)
• Retrieved documents (4 chunks)
• Summarized previous findings (1 summary)
• Current request (1 turn)

All of these compete for the same finite token budget. Context engineering decides what stays, what goes, what gets summarized, and what gets discarded.

This distinction matters because as agents run longer, the wording of the original prompt matters less than the ongoing curation of what surrounds it turn after turn. A perfectly worded system prompt ("You are a helpful research assistant...") can still produce a confused agent if the context around it, ten steps in, is cluttered with stale tool results and irrelevant history.

Research from Anthropic on long-context models also shows a phenomenon called "lost in the middle" (Liu et al., 2023): models pay less attention to information in the middle of long contexts, and more to the beginning and end. This means dumping everything into context and hoping it all gets equal weight is not a good strategy; you have to actively curate.

How context engineering differs in practice

The table below illustrates the difference between prompt engineering and context engineering across several dimensions:

| Dimension | Prompt Engineering | Context Engineering | |-----------|-------------------|----------------------| | Scope | Single message wording | Entire context across all turns | | Decision point | Before agent starts | Every turn during execution | | What changes | System message phrasing | What data is included/excluded | | Typical effort | One-time tuning | Continuous curation | | Failure mode | Unclear instructions | Context bloat, attention dilution | | Success metric | Single-turn response quality | Multi-turn agent reasoning consistency | | Cost impact | Minimal | Significant (scales with run length) |

For a 50-turn agent run researching competitive markets, prompt engineering might get you 70% of the way to quality, but context engineering gets you from 70% to 95%. The initial prompt is important; maintaining a clean, focused context throughout the run is critical.

Five layers, one shared budget

Every agent turn assembles its context from roughly the same five sources:

| Layer | Typical cost | Fixed or growing? | |-------|----------|-----------| | System prompt | 1,000-2,000 tokens | Fixed | | Current request | 50-200 tokens | Per turn | | Retrieved documents (RAG) | 1,000-3,000 tokens | Per turn, fresh retrieval | | Conversation history | 200-500 tokens per turn | Growing with run length | | Tool results | 200-5,000 tokens per result | Growing with tool calls |

In a 20-turn agent run:

• System prompt: 1,500 tokens (fixed)
• Current request: 100 tokens
• Retrieved docs: 2,000 tokens
• History from previous 19 turns: 19 × 300 = 5,700 tokens (grows linearly)
• Tool results: 19 × 800 = 15,200 tokens (grows linearly)

Total: 1,500 + 100 + 2,000 + 5,700 + 15,200 = 24,500 tokens

In a 100-turn agent run (same technique, no optimization):

• System: 1,500
• Current: 100
• Retrieved: 2,000
• History: 99 × 300 = 29,700 (now dominates)
• Tool results: 99 × 800 = 79,200 (way dominates)

Total: 112,500 tokens -- you've hit your context window limit, and the system has to truncate, losing the early turns' context.

The lesson: history and tool results grow linearly with run length. Without curation, they dominate the context window.

Three techniques for keeping context lean over a long run

Technique 1: Summarize instead of accumulating

Rather than keeping every tool result verbatim across a twenty-step run, periodically compress older results into a short summary that preserves the decision-relevant facts and drops the rest.

def compress_old_results(context_history, keep_last_n_turns=3):
    """Keep recent results verbatim, summarize older ones."""
    if len(context_history) <= keep_last_n_turns:
        return context_history  # Nothing to compress

    old_results = context_history[:-keep_last_n_turns]
    recent_results = context_history[-keep_last_n_turns:]

    # Compress old results into a summary
    summary = summarize_results(old_results)

    return [summary] + recent_results

# In your agent loop:
# Every 5 turns, compress the history
if step % 5 == 0:
    context = compress_old_results(context)

Example: instead of keeping a 10-page raw web page dump (5,000 tokens), extract and keep only the 3 key facts (100 tokens). The agent has everything it needs without the noise.

Technique 2: Scope retrieval to the current step, not the whole task

Retrieve fresh, narrowly-targeted context for what the agent needs right now, rather than retrieving broadly once at the start and carrying that same large context through every subsequent step.

def retrieve_for_current_step(current_goal, vector_db, budget=1000):
    """Retrieve context specific to this step only."""
    # Don't retrieve "everything about the topic"
    # Retrieve "what I need right now"

    retrieved = vector_db.search(
        query=current_goal,  # Specific goal, not broad task
        top_k=5,
        max_tokens=budget
    )
    return retrieved

# In agent loop:
# Each turn, retrieve fresh, relevant-to-this-step context
for step in range(num_steps):
    current_step_goal = "I need to find X for this step"
    context_for_this_step = retrieve_for_current_step(
        current_step_goal,
        vector_db
    )
    # Use context_for_this_step, discard after this step

This keeps context tight: turn 10 has context about turn 10's goal, not a stale accumulation of turns 1-9's retrieved docs.

Technique 3: Give tools a compact default output

Design tools to return a short, structured summary by default, with full detail available on a follow-up call only if the agent actually needs it. This keeps a single verbose tool result from crowding out everything else in context.

class WebSearchResult:
    def __init__(self, title, url, summary, full_text):
        self.title = title
        self.url = url
        self.summary = summary       # ~200 tokens
        self.full_text = full_text   # ~5000 tokens

    def __str__(self):
        # When used in context, return compact version
        return f"{self.title} ({self.url}): {self.summary}"

def web_search(query: str):
    """Search the web, return compact by default."""
    results = search_api(query)

    return [
        WebSearchResult(
            title=r['title'],
            url=r['url'],
            summary=extract_key_sentences(r['text']),  # 2-3 key sentences
            full_text=r['text']
        )
        for r in results[:5]
    ]

# In agent context:
# The agent sees compact results
results = web_search("who won the 2024 olympics")
print(results[0])
# Output: "2024 Olympics Results (url): Here are the major medal winners..."

# If the agent needs full detail, it calls a separate tool
def get_full_article(result_index: int):
    # Return full_text for a previous search result
    pass

The agent sees search results as short summaries by default (~150 tokens total). If it needs more detail, it explicitly requests it. This keeps context lean for the happy path.

A long run, with and without curation

Scenario: An agent researches a competitive landscape across 20 steps.

Without curation:

Turn 1-5: Agent searches for competitors A, B, C. Each search returns 5 results, each with a full page dump (5,000 tokens). Context is accumulating.

Turn 10: Context now has 50 full pages (250,000 tokens) crammed into the context window. Most are irrelevant (navigation menus, ads, outdated pricing). The model's turn-10 decision has to pick out the 2-3 facts that matter from 250,000 tokens of noise.

Turn 15: The framework truncates; early turns (which might have had the original goal and requirements) are dropped. The model on turn 15 has forgotten what it was supposed to be looking for.

Turn 20: Agent finishes, but likely made wrong decisions mid-run because relevant signal was buried in noise. Output is mediocre.

Context progression: 0 → 10K → 30K → 80K → 150K → ??? (truncation happens)

With curation (same task, same 20 steps):

Turn 1-5: Agent searches, retrieves results. After each search, compresses findings into a bulleted summary (100 tokens): "Competitor A: pricing $50/month, main features are X and Y, customer reviews focus on reliability."

Turn 10: Context has the original goal (500 tokens) + compressed findings from A, B, C (300 tokens total) + current search in progress (1000 tokens) = 1,800 tokens. Everything is relevant; nothing is noise.

Turn 15: Same; context remains small and focused.

Turn 20: Agent finishes with excellent decision quality because the context was always signal-rich.

Context progression: 0 → 2K → 3K → 3.5K → 4K (stays small throughout)

The difference: without curation, raw growth. With curation, stable, manageable context that stays relevant. Same task, same research, vastly different outcomes.

Context curation in practice: A case study

Company: TechReview Corp, a SaaS comparison service

Problem: An agent crawled competitor websites, read product documentation, and analyzed pricing to generate comparison reports. After 25 API calls and document retrievals across a single task, the agent's context window (128K tokens) was exhausted. The agent started hallucinating pricing and features because relevant information was truncated.

Solution: Implement a three-layer memory strategy:

1. Immediate layer (current turn): Keep only the active comparison being written (500 tokens)
2. Active layer (last 5 turns): Keep recent findings in verbatim form (2,000 tokens)
3. Archived layer (older turns): Compress into structured facts (1,000 tokens): "Product A: $50/user/month, supports REST API, 99.95% SLA"

Implementation:

class ComparisonAgent:
    def __init__(self, max_context_tokens=100000):
        self.immediate = []       # Current turn only
        self.active = []          # Last 5 turns
        self.archive_summary = "" # Everything older, compressed
        self.max_context = max_context_tokens

    def add_finding(self, finding: str, source: str):
        """Add a new finding and manage layers."""
        self.immediate.append({"finding": finding, "source": source})

        # Count tokens (simplified: 4 chars ≈ 1 token)
        current_tokens = self._estimate_tokens()

        if current_tokens > self.max_context * 0.8:
            # Time to promote old findings to archive
            self._promote_to_archive()

    def _promote_to_archive(self):
        """Move oldest findings to compressed archive."""
        if len(self.active) > 5:
            to_archive = self.active[:-5]
            # Call model to compress
            compressed = self._compress_findings(to_archive)
            self.archive_summary += f"\n{compressed}"
            self.active = self.active[-5:]

    def _estimate_tokens(self) -> int:
        """Estimate total context tokens used."""
        immediate_tokens = len(str(self.immediate)) // 4
        active_tokens = len(str(self.active)) // 4
        archive_tokens = len(self.archive_summary) // 4
        return immediate_tokens + active_tokens + archive_tokens

    def build_context(self) -> str:
        """Assemble context for the next turn."""
        context = ""

        if self.archive_summary:
            context += f"## Archived findings (compressed):\n{self.archive_summary}\n\n"

        if self.active:
            context += "## Recent findings:\n"
            for item in self.active:
                context += f"- {item['finding']} (source: {item['source']})\n"
            context += "\n"

        context += "## Current focus:\n"
        for item in self.immediate:
            context += f"- {item['finding']}\n"

        return context

Result: The agent successfully generated detailed comparison reports across 35+ sources without truncation or hallucination. Context size stayed under 90K tokens throughout, with fresh information always accessible.

Edge case: The lost-in-the-middle trap

A counterintuitive problem: even with a large context window, placing crucial facts in the middle of the context can reduce model attention to them. A research paper (Liu et al., 2023) showed that models performing retrieval tasks accuracy was highest for facts at the beginning and end of context, and degraded for facts in the middle.

Implication for context engineering: Don't just truncate old context; strategically place important facts. A good pattern:

1. Beginning of context: System prompt + current goal / task
2. Middle: Supporting facts, recent history, retrieved documents
3. End: The specific current turn, immediate problem to solve

Avoid placing critical constraints or the original requirements in the middle of a 50K-token context. Repeat them at the beginning or end if needed.

def build_ranked_context(
    goal: str,
    system_role: str,
    retrieved_docs: list[str],
    conversation_history: list[str],
    current_question: str
) -> str:
    """Build context with strategic placement."""

    # Priority 1: Beginning (highest attention)
    start = f"""Role: {system_role}

Original goal: {goal}

CRITICAL CONSTRAINTS:
- Stay factual; never hallucinate pricing or features
- Flag any uncertainty
- Cite sources explicitly
"""

    # Priority 2: Middle (supporting context)
    middle = f"""## Retrieved documentation
{chr(10).join(retrieved_docs[:5])}

## Recent conversation
{chr(10).join(conversation_history[-5:])}
"""

    # Priority 3: End (current focus, high attention)
    end = f"""## Immediate task
{current_question}

REMINDER: Goal is {goal}
Focus on accuracy over completeness."""

    return f"{start}\n\n{middle}\n\n{end}"

This ensures critical information is at the boundaries where model attention is strongest.

Why larger context windows don't solve this

You might think "let's just use a model with a 200,000-token window, then we don't have to worry about curation."

Research from Anthropic's own papers on long-context models shows this reasoning is flawed:

1. "Lost in the middle" effect: Models pay less attention to information in the middle of long contexts. If your important facts are in the middle of 200,000 tokens, the model might not attend to them as strongly as facts at the beginning or end.

2. Attention dilution: Every extra token of irrelevant context spreads the model's attention thinner. A decision that should be made with 5K tokens of focused context performs worse when it's actually made with 50K tokens of noise + 5K tokens of signal.

3. Cost compounds: A larger context window doesn't reduce the cost of processing irrelevant context; it multiplies it. You're paying for every token, including the noise.

A 128,000-token context window with ruthless curation often outperforms a 200,000-token window without curation.

Common mistake

Assuming that because a model has a large context window, filling it generously is harmless. Anthropic's own guidance points out that irrelevant or poorly organized context can measurably degrade an agent's output quality well before the window's token limit is reached. Context engineering is about curating for relevance and clarity, not just staying under a token count.

The analogy: a person solving a complex problem doesn't benefit from having every research paper ever written about the topic nearby. They benefit from having the three most relevant papers and a clear summary of what they've already learned. Context engineering applies the same principle: design your agent's context to be a well-curated briefing, not a dump of everything that could possibly be relevant.