The Architecture Stack Under an Agent

The layers below your agent code

Most agent tutorials focus on the "orchestrator" -- the loop that calls the model, routes tool calls, and checks stopping conditions. But that orchestrator is sitting on top of an entire infrastructure stack. Understanding those layers is crucial because a bug in any of them can break the agent, and optimization often happens at these lower layers, not in the agent logic itself.

The four main layers below the agent orchestrator are:

1. Model serving -- how the LLM itself is accessed and managed
2. Tool execution -- how tool calls are routed and executed
3. State and memory -- how results are stored and retrieved between steps
4. Safety and governance -- how guardrails and permissions are enforced

Layer 1: Model serving

The agent calls a language model at every step. How that model is accessed shapes the entire agent's latency, cost, and reliability.

Option A: Hosted API (e.g. Claude API, OpenAI API)

The simplest approach is to call a hosted API. Each time the orchestrator wants to call the model, it sends an HTTP request to Anthropic or another provider's endpoint.

Advantages:

• No infrastructure to maintain; the provider handles scaling, failover, and version management
• Immediate access to new model versions
• Transparent pricing and usage tracking

Disadvantages:

• Network latency: each step incurs a round trip
• Rate limits: high-volume agents may hit API quotas and need queuing logic
• Cost: every model call is metered and charged, even for short (1,000-token) calls

When to use: Most applications. The simplicity and reliability outweigh the cost for most agent tasks. This is Anthropic's own recommendation for teams building agents.

Option B: Self-hosted model

An organization can run an LLM on its own infrastructure (on-prem servers or rented cloud instances), using a serving framework like vLLM or Ray Serve.

Advantages:

• Lower per-token cost at scale (buy hardware once, run models repeatedly)
• No external API calls; latency is lower if the server is geographically close
• Full control over model versions and configurations

Disadvantages:

• Infrastructure cost: GPU clusters are expensive to run and maintain
• Complexity: orchestrating, scaling, and updating the serving infrastructure is non-trivial
• Operational burden: the team owns availability, patching, and troubleshooting

When to use: Only if you have high-volume, latency-sensitive agents and the ROI of maintaining infrastructure exceeds the cost of API calls. Most organizations should use hosted APIs.

Serving architecture comparison: illustrative estimates

The following table compares the two approaches across key operational dimensions. Note: these are illustrative estimates based on typical production scenarios; actual costs and latencies vary significantly based on scale, region, model size, and provider pricing.

| Dimension | Hosted API | Self-hosted cluster | |-----------|-----------|-------------------| | Setup time | Hours (configure credentials, rate limits) | Weeks (procure hardware, configure orchestration) | | Monthly cost at 1M requests/month | ~$1,200–$1,800 (depending on model) | ~$2,000–$4,000 (GPU depreciation + ops) | | Monthly cost at 100M requests/month | ~$120,000–$180,000 | ~$8,000–$15,000 (amortized hardware) | | Breakeven request volume | N/A | ~50M requests/month (varies by setup) | | Latency per call | 200–500ms (network round trip) | 100–300ms (local network) | | Availability/SLA | Provider's SLA (~99.9%) | Your responsibility; typically 95–99% | | Token context support | Latest versions (up to 200k) | Whatever you deploy | | Operations staff needed | ~0.1 FTE (monitoring only) | ~1–2 FTE (24/7 oncall) |

The breakeven point is real: below ~50M requests/month, hosted APIs are cheaper and simpler. Above that, self-hosted becomes economically viable, but only if you have operations expertise.

Model serving architecture details

Within either option, the model serving layer handles several concerns:

Request batching: If multiple agents need to call the model at the same time, a well-designed serving layer batches them together so the model runs one optimized batch inference instead of multiple separate inferences. This reduces total latency and hardware utilization.

Token streaming vs. full response: A serving layer can return the full response at once, or stream tokens as they're generated. Streaming reduces latency for long responses but complicates orchestrator logic (the agent can't make a decision until the full response arrives, so streaming is most useful in UIs, less so in agent loops).

Context window management: Very long agent contexts might exceed the model's context window. The serving layer (or orchestrator) needs to handle this: truncate old history, summarize earlier results, or split the context across multiple requests.

Real-world case study: a fintech company's serving layer crisis

A financial services startup built an agent to summarize loan documents for underwriters. Early on, they used the hosted API, and at low volume (~100 documents/day) it worked fine. As the product took off, they were running 10,000 loan summaries per day, and their API bill had become their largest infrastructure cost. The team decided to self-host a deployment using vLLM on a 4-GPU cluster.

Within months, they hit a problem: concurrent requests for multiple underwriters would overwhelm the batch queue, causing 30-second delays. The serving layer wasn't equipped to handle mixed request sizes (short summaries vs. long analytical reports). They had to add separate queues by document size, and implement priority handling so urgent underwriter requests jumped the queue. Even then, GPU utilization was uneven — sometimes one GPU was bottlenecked while others sat idle.

The lesson: self-hosting solved the cost problem but surfaced the complexity of the serving layer. They eventually settled on a hybrid: for high-volume, latency-sensitive underwriting summaries, they kept self-hosted infrastructure; for ad-hoc exploration tools used by analysts, they used the hosted API. The breakeven analysis was right, but only after accounting for the staffing cost of maintaining the serving layer and the opportunity cost of delays.

Layer 2: Tool execution

When the model decides to call a tool, that tool call has to be executed. A simple orchestrator might execute tools synchronously (wait for the tool to finish, then call the model again). A sophisticated one might queue or parallelize them.

Synchronous tool execution

# Simple: one tool call at a time
context = [goal]
while not done:
    response = call_model(context)
    if response.is_final_answer:
        return response

    tool_result = execute_tool(response.tool_call)  # Wait for it to finish
    context.append(tool_result)

This is straightforward but becomes slow if tools are slow. If a tool takes 5 seconds (e.g., a database query or API call), the agent waits 5 seconds per iteration. An agent that takes 10 iterations will wait 50+ seconds just for tool execution.

Asynchronous / parallel tool execution

Some agents can call multiple tools in a single iteration:

# More complex: call multiple tools, wait for all
context = [goal]
while not done:
    response = call_model(context)
    if response.is_final_answer:
        return response

    # Execute multiple tools in parallel
    tool_results = await asyncio.gather(
        execute_tool(tool_call1),
        execute_tool(tool_call2)
    )
    context.extend(tool_results)

This requires the orchestrator to handle the fact that the model might not have been designed to output multiple tool calls in one response. Some models (like Claude) are fine-tuned to handle this; others are not.

Tool execution with timeout and retry logic

Real-world tools fail unpredictably. A production tool execution layer needs to handle timeouts, transient errors, and retries:

import asyncio
from datetime import datetime

async def execute_tool_with_resilience(tool_call, max_retries=2, timeout_sec=10):
    """
    Execute a tool with timeout and exponential backoff retry logic.
    Returns a structured result with metadata for the agent.
    """
    for attempt in range(max_retries):
        try:
            # Set a timeout for this tool call
            result = await asyncio.wait_for(
                execute_tool(tool_call),
                timeout=timeout_sec
            )
            return {
                "status": "success",
                "content": result,
                "attempt": attempt + 1,
                "timestamp": datetime.now().isoformat()
            }
        except asyncio.TimeoutError:
            if attempt < max_retries - 1:
                # Wait exponentially longer before retrying (1s, 2s, 4s...)
                wait_time = 2 ** attempt
                await asyncio.sleep(wait_time)
                continue
            else:
                return {
                    "status": "timeout",
                    "error": f"Tool did not respond within {timeout_sec}s after {max_retries} attempts",
                    "timestamp": datetime.now().isoformat()
                }
        except Exception as e:
            return {
                "status": "error",
                "error": str(e),
                "attempt": attempt + 1,
                "timestamp": datetime.now().isoformat()
            }

The agent can check the status field and decide whether to retry differently, log the failure, or take an alternative action.

Tool routing and permissions

The tool execution layer also enforces permissions. If the model calls delete_database(), the execution layer should check: is the agent allowed to call this tool? In what context? With what arguments? These checks happen at the tool execution layer before the tool actually runs.

def execute_tool(tool_call: ToolCall, user_context: dict) -> str:
    # Check permissions first, using the current user's context
    if not check_permission(
        user_context,
        tool_call.name,
        tool_call.args
    ):
        return "Error: permission denied"

    # Validate arguments match tool schema
    if not validate_tool_args(tool_call.name, tool_call.args):
        return "Error: invalid arguments"

    # Then execute
    tool_func = tools[tool_call.name]
    try:
        result = tool_func(**tool_call.args)
        return result
    except Exception as e:
        return f"Error: {e}"

This is where guardrails live. A well-designed system never lets an unapproved tool call reach the actual tool. The permission check should be context-aware: even if a user can normally call a tool, there may be specific arguments (e.g., querying customer data from a different department) that require additional approval.

Layer 3: State and memory

Between iterations, the agent's context needs to be stored somewhere. Early iterations append results to a list in memory; later iterations need to fetch them back. For agents that run across multiple user sessions or survive a server restart, that state needs to be persisted.

State persistence tradeoffs: illustrative estimates

Different state backends have different consistency, latency, and cost characteristics. The following table shows typical illustrative scenarios:

| Backend | Latency per read | Latency per write | Durability | Cost at 1M iterations/day | Scaling complexity | |---------|-----|-----|-----------|------------|-----------------| | In-memory (Python dict) | <1ms | <1ms | None (lost on crash) | ~$0 | Very low; single-machine only | | Redis cache | 5–15ms | 5–15ms | Optional (AOF/RDB) | ~$30–80/mo | Low; handles 100k ops/sec | | PostgreSQL | 10–50ms | 50–100ms | Full (ACID) | ~$100–300/mo | Medium; requires connection pooling | | DynamoDB | 5–20ms | 5–20ms | Full (AWS SLA) | ~$50–500/mo | High (pay per throughput) | | MongoDB | 5–20ms | 10–50ms | Configurable | ~$50–200/mo | Medium (replication/sharding) |

For a typical agent doing 100,000 iterations per day with an average state size of 5KB: Redis hits the throughput ceiling around 500 concurrent agents; PostgreSQL can handle 2,000–5,000 agents; DynamoDB scales horizontally but with higher per-operation cost. The choice depends on whether you optimize for latency (Redis), cost (PostgreSQL), or scale (DynamoDB).

In-memory state

For a short-lived agent (one user, one session, runs for a few minutes), keeping state in Python objects is fine:

class AgentRun:
    def __init__(self, goal: str):
        self.goal = goal
        self.context = [goal]  # Grows as iterations happen
        self.tool_results = []  # Stores all results
        self.turn_count = 0

agent = AgentRun("Summarize my emails")
# Agent loop modifies agent.context, agent.tool_results

This works for single-threaded, short-lived agents, but breaks as soon as you need to:

• Run the same agent across multiple requests (context is lost when the process restarts)
• Run multiple agents in parallel (they might overwrite each other's state)
• Resume an agent that was interrupted (no checkpoint to resume from)

Persistent state

Production agents store state in a database or cache layer:

# Pseudocode: persist state to a database
def run_agent_step(agent_id: str, goal: str):
    # Fetch prior state
    run = fetch_agent_run(agent_id)  # From database
    if not run:
        run = create_agent_run(agent_id, goal)
        save_agent_run(run)

    # Run one step
    response = call_model(context=run.context)

    # Persist updated state
    run.context.append(response)
    run.turn_count += 1
    save_agent_run(run)  # Write back to database

    # Continue or finish
    if response.is_final_answer:
        return response
    else:
        # Queue the next step (or have the orchestrator loop and call again)
        return {"status": "in_progress", "next_step_queued": True}

This allows the agent to:

• Survive process restarts (state is in the database, not lost)
• Be resumed from an interrupt (fetch the run from the database, pick up where it left off)
• Support long-running tasks (agent runs across multiple API calls, without keeping the connection open the whole time)

Tradeoff: latency vs. durability

Writing to a database on every iteration adds latency (database writes are slower than in-memory appends). An optimization: batch multiple iterations in memory, then persist periodically or at the end. Another tradeoff: if you persist every 5 iterations, a crash loses up to 5 iterations of work.

Edge case: state corruption and recovery

A subtle problem: what if the state persisted to the database is partially corrupted, or the write fails halfway through? A real production system needs transaction semantics:

def persist_agent_step(agent_id: str, step_data: dict):
    """
    Atomically persist an agent step. If anything fails, the entire
    transaction rolls back (no partial updates).
    """
    with db.transaction():
        # Fetch the current state (with row lock if using PostgreSQL)
        run = db.query("SELECT * FROM agent_runs WHERE id = %s FOR UPDATE", agent_id)

        # Check optimistic lock: did anyone else update this between our read and write?
        if run.version != step_data.expected_version:
            raise StaleStateError("State was modified by another request")

        # Update atomically
        db.query("""
            UPDATE agent_runs
            SET context = %s, turn_count = %s, version = version + 1, updated_at = NOW()
            WHERE id = %s AND version = %s
        """, step_data.context, step_data.turn_count, agent_id, step_data.expected_version)

        # If we get here, the write succeeded atomically
        return True

Without atomicity, a crash or race condition could leave the database with inconsistent state: the context updated but the turn count not incremented, or vice versa. The agent would resume in an undefined state.

Layer 4: Safety and governance

Before an agent can do anything (call a tool, make a decision), it should pass through safety layers.

Permission checks

The simplest layer: does the user have permission to run this agent? Does the agent have permission to call this tool with these arguments?

def run_agent(user_id: str, agent_id: str):
    agent = fetch_agent(agent_id)

    # Check user permission
    if user_id not in agent.authorized_users:
        raise PermissionError(f"User {user_id} not authorized")

    # Check agent's tool permissions
    if agent.permissions.deny_tools:
        # Agent is restricted from calling certain tools
        pass

    return orchestrate_agent(agent)

Input validation

Before the orchestrator runs, check the user's input (goal) and agent configuration for red flags:

• Is the goal reasonable? Does it contain malicious instructions or attempts to break the agent's constraints?
• Is the agent's configuration valid? Does it reference tools that don't exist?

This is where guardrail models (a second, lightweight model call) sometimes help. Before running the expensive agent loop, a quick sanity check.

Audit and logging

Every significant action should be logged for auditability: which user ran which agent, what tools it called, what results it saw, and what final output it produced. This is essential for debugging and compliance.

def run_agent_step(user_id: str, agent_id: str, step: int):
    # ... run the step ...
    log_action(
        user_id=user_id,
        agent_id=agent_id,
        step=step,
        model_input=context,
        tool_calls=response.tool_calls,
        tool_results=tool_results,
        timestamp=now()
    )

How the layers interact: a trace

To see how all four layers work together, consider a simple agent run:

User initiates: POST /agents/my-agent/run with goal "List my top 5 projects by priority"

Layer 4 check: Permission validator checks: does user "alice" have permission to run agent "my-agent"? Yes. Is the goal safe? Yes. Proceed.

Layer 3 setup: State manager creates a new AgentRun record in the database with id "run-12345". Stores goal and empty context.

Iteration 1:

Layer 3 fetch: State manager fetches run "run-12345". Context is just the goal.

Layer 1 call: Model serving receives the context and calls Claude. Response: "I need to fetch the user's projects first."

Layer 2 execute: Tool executor checks: is agent allowed to call fetch_projects? Yes. Calls it. Returns list of 10 projects with priorities.

Layer 3 save: State manager appends tool result to context, increments turn_count, saves back to database.

Layer 4 log: Audit log records: user alice, agent my-agent, iteration 1, tool fetch_projects, got 10 results.

Iteration 2:

Layer 3 fetch: Context now has goal + project list.

Layer 1 call: Model responds: "I've retrieved the projects. The top 5 by priority are..."

Layer 4 log: Audit log records the final response.

Return: Response sent to user. AgentRun marked complete in database.

If the user later wants to resume or review what happened, all of that information is in the database and logs.

Hosted agents vs. custom infrastructure

Most teams should use a hosted agent platform or API (like Anthropic's native agent capabilities, LangGraph Cloud, or similar). This abstracts away layers 1-4. You write the agent logic; the platform handles serving, tool routing, state persistence, and safety.

The alternative is building your own orchestrator and managing the four layers yourself. This is necessary if:

• You need highly customized tool behavior
• You need specific compliance or data residency requirements
• You're building a platform that others will build agents on top of

For most applications building one or two agents, the hosted approach is simpler, cheaper, and more reliable.

Common mistake

Focusing entirely on the orchestrator logic (which pattern to use, which stopping conditions) while ignoring the infrastructure layers. An agent with brilliant loop logic but a fragile state layer will lose work when it crashes. An agent with perfect tool permissions but no error handling in the serving layer will fail mysteriously on network timeouts. Spend time on all four layers, not just the orchestrator itself.