LLM Foundations Every Agent Builder Needs

Everything is next-token prediction

At its core, a large language model does one thing: given a sequence of tokens, predict the most likely next token, then repeat. Chat, tool calls, and code generation are all this same mechanism applied to differently formatted input -- a tool call is just a specially formatted piece of text the model has learned to produce when it decides an action is needed, not a fundamentally different capability bolted on.

This matters for agent builders because it reframes what a "tool call" is: it's not the model reaching outside itself, it's the model predicting text that your orchestrator recognizes and executes on the model's behalf. If your tool-calling format is inconsistent or your tool descriptions are confusing, you are making next-token prediction harder, not giving the model a fundamentally different capability to lean on.

The transformer, briefly

Modern LLMs are built on the transformer architecture, introduced in the 2017 paper "Attention Is All You Need." Its key mechanism, self-attention, lets the model weigh the relevance of every other token in its context when producing each new token -- which is why longer, well-organized context can meaningfully improve output quality (the model can actually "see" the relevant parts) and why irrelevant or contradictory context can hurt it (the model has to weigh noise alongside signal).

Three training stages behind a chat model

Why this matters for prompt and tool design

Because the model was fine-tuned on specific formats for instructions and tool calls, agent builders get more reliable behavior by working with those learned formats -- using the documented system-prompt and tool-schema conventions for the model you're using -- rather than inventing a custom format the model hasn't specifically been trained to handle well. This is also why "the same prompt behaves differently on a different model": each model's fine-tuning and alignment stage shaped its specific expectations and defaults.

Next-token prediction, made concrete

It helps to see what "predict the next token" actually looks like mechanically. Given the input "The weather in Boston is currently", the model doesn't retrieve a stored fact -- it computes a probability distribution over every possible next token given everything before it, and picks (or samples) one, say "68". It then treats "The weather in Boston is currently 68" as the new input and repeats the exact same operation to produce the next token, and the next, one at a time, until it produces a token that signals the response is complete. A tool call is generated by this identical mechanism: the model has learned, from its fine-tuning examples, that certain kinds of requests are followed by a structured pattern like {"tool": "search", "query": "..."}, and it predicts those tokens the same way it predicts ordinary words -- there is no separate "decide to use a tool" subsystem running underneath.

Self-attention and why context organization matters

Self-attention is the mechanism that lets the transformer weigh every other token in context when producing the next one -- but "weigh" is not uniform. In practice, models attend more reliably to information that is well-organized, clearly labeled, and free of contradictions than to the same facts buried in noisy or redundant text, even though both technically fit within the context window. This is the direct link between this lesson and the later lesson on context engineering: a transformer's core mechanism is why curating context (not just fitting it under a token limit) measurably changes output quality.

Code example: how training stages shape behavior

Here's a concrete example showing how fine-tuning affects tool behavior:

# Without fine-tuning (raw next-token prediction):
# Model sees "Tool available: get_weather(city)" in a raw text document
# Probability of outputting a structured tool call is low (~5%)
# Model might output prose: "I should check the weather"

# After supervised fine-tuning on tool examples:
# Model sees the same tool definition in Anthropic's format
# Probability of structured tool call is high (~95%)
# Model outputs: {"type": "tool_call", "name": "get_weather", "arguments": {"city": "Boston"}}

# The mechanism is identical (next-token prediction)
# But the fine-tuning changed what the model learned to predict in this context

This is why using the exact format recommended by your model provider matters so much: the model's fine-tuning stage was the only opportunity to train it on your format, and that training already happened.

Why token limits matter for agents

Agents see growing context as the loop runs: goal → first tool result → second tool result → etc. Each iteration adds tokens to the context. If an agent's context window is 8,000 tokens and the loop runs five times with 1,500-token tool results each, the context grows by 7,500 tokens. By turn 5, context might be nearing the limit, and the model's ability to attend to earlier information degrades. This is why agents with longer context windows (200k tokens and beyond) are more flexible -- they can loop longer without context compression, and they can carry richer context through the loop.

Example: An agent researching a complex topic might retrieve several long documents (each 2,000-4,000 tokens). A model with a 4,000-token context window can't hold all of them simultaneously. A model with a 200k-token context can ingest all at once and see all the relationships. Same agent, same tools, radically different capability -- because of the underlying model's context window, not the agent's ingenuity.

Model capability vs. context window tradeoff table

The following table (illustrative estimates) shows how different models balance capability with context window size:

| Model | Context window | Typical latency | Cost per 1M tokens | Recommended for agents with | |-------|---------|---------|---------|---------| | Claude 3 Haiku | 200k tokens | 200–300ms | ~$0.25–0.30 | Simple tools, short loops, cost-sensitive | | Claude 3.5 Sonnet | 200k tokens | 400–600ms | ~$2.00–2.50 | Most production agents (best all-rounder) | | Claude 3 Opus | 200k tokens | 800–1,200ms | ~$15.00 | Complex multi-step reasoning, large tools | | GPT-4 Turbo | 128k tokens | 500–800ms | ~$15.00 | OpenAI-specific workflows | | Llama 3.1 (70B) | 128k tokens | 1,500–3,000ms | ~$0.50 (self-hosted) | High volume with self-hosted infra |

A rule of thumb: Sonnet is the default starting choice for agents. Haiku works for simple tasks at 1/10th the cost. Opus handles the most complex reasoning. The key insight is that context window alone doesn't predict agent capability; a Haiku model with efficient prompts and good tool design often outperforms an Opus model with bloated context.

Tokenization: why it matters for tool calls

Tokens are not words. The word "hamburger" might be one token; the symbol "😀" might be multiple tokens or not tokenize well at all. Tool-call schemas that are simple and use common characters (like JSON with {"tool": "name", "args": {}}) tokenize more efficiently than complex or unusual formats. An inefficient format might use 20% more tokens for the same information, which means either shorter context, higher costs, or both.

This is why agent builders prefer structured, common formats like JSON for tool calls rather than custom formats like "CALL tool_name WITH arg1=X arg2=Y" -- the model was trained to output JSON thousands of times, it tokenizes efficiently, and it parses reliably on the receiving end.

The relationship between model capability and agent design

A larger, more capable model (like Claude 3.5 Sonnet) can handle:

• More complex tool sets (more tools to choose from at each step)
• Longer context windows without degradation (richer memory)
• More nuanced stopping conditions (it can recognize more subtle cues for when to stop)
• Better tool use (it reads long tool descriptions accurately and uses them correctly)

This doesn't mean a bigger model is always the answer, but it does mean architecture choices that work for a 70B model might fail on a 7B model. An agent design that spams the model with dozens of tool options might work fine on a very large model but cause a smaller one to get confused. Similarly, a simple single-turn prompt that works for a powerful model might need an agent loop (multiple turns) to work reliably on a smaller model.

How fine-tuning shapes tool behavior

During the supervised fine-tuning stage, the model is trained on examples of how to respond to tool calls. If a model is fine-tuned on thousands of examples where tool calls are formatted as JSON with {"type": "tool_call", "name": "...", "parameters": {...}}, it will reliably produce that format when asked to use a tool. If it's fine-tuned on examples where the format is different, it will produce that format instead.

This is why using the exact format recommended by the model's provider is so important. Claude models are fine-tuned on Anthropic's specific tool-use format; GPT models are fine-tuned on OpenAI's format. Anthropic's engineering guidance is not "you could use any format," it's "use the format Claude was trained on for best reliability."

A concrete example: context and tool reliability

Consider an agent that must call a tool with the correct arguments. The model is given a tool description:

Tool: update_customer_billing
Description: Updates a customer's billing address or payment method
Parameters:
  - customer_id (string, required): The numeric ID of the customer, never a name
  - billing_address (object, optional): New address {street, city, state, zip}
  - payment_method (object, optional): New payment method {type, token, expiry}
  - notify_customer (boolean, optional, default=True): Send confirmation email

A smaller model might see this description and later call update_customer_billing(customer_name="Alice Smith", ...) -- it misread the requirement that customer_id is numeric. A larger model reads more carefully and sends customer_id="12345" correctly. This is not a philosophical difference in the models' "understanding"; it's a result of the larger model's superior ability to track details across longer, more complex context.

An agent builder can compensate by clarifying the tool description even further, using a different format, or by adding a validation step that checks the model's tool call before executing it. But the underlying constraint is real: tool use fidelity is partly a function of model capability.

Real-world case study: a fintech startup's model selection dilemma

A fintech startup built a transaction analysis agent using GPT-3.5-turbo. The agent worked well for simple queries but started failing on complex questions like "What are all my transactions in the 'Utilities' category from Q3 last year, excluding one-time charges?" The model would call tools with malformed filters, ask for the wrong date ranges, or omit critical arguments.

They tried three approaches:

1. Better prompting (~10% improvement): Adding more details to tool descriptions helped slightly, but the core problem remained.
2. Switching to GPT-4 (~40% improvement): The larger model read instructions more carefully and made fewer argument errors.
3. Switching to Claude 3.5 Sonnet (~50% improvement): Different architecture + better fine-tuning for tool use meant fewer errors and better handling of complex nested filters.
4. Validation + retry (with any model): Before executing a tool call, validate arguments. If invalid, send feedback to the model and ask it to retry. This generic approach improved any model by ~25–35%.

The lesson: model capability matters for agents, but is one of several levers you can pull. Validation and retry logic can sometimes substitute for a larger model, but they add latency and cost. For critical paths, the right model + validation is the winning combination.

Edge case: format brittleness across model versions

A subtle issue: model versions sometimes change the expected tool-call format, or stop supporting an older format. For example, a model might be fine-tuned to output XML-style tool calls:

<tool_call>
  <name>get_user_info</name>
  <arguments>
    <user_id>12345</user_id>
  </arguments>
</tool_call>

In the next version, the model switches to JSON:

{"tool_call": {"name": "get_user_info", "arguments": {"user_id": 12345}}}

If your parsing logic is hardcoded for XML, the newer model version breaks your agent silently: the model outputs JSON, your parser can't find XML, and the agent stops calling tools. The fix: always support multiple formats in your parser, or query the model's documentation before upgrading to see if the format changed.

Common mistake

Assuming that because a model can theoretically process any text format, all formats work equally well for instructions or tool calls. In practice, models are noticeably more reliable when given tool definitions and instructions in the format their provider's documentation recommends, because that is the format the model actually saw thousands of times during fine-tuning. Don't invent a new tool-call format "to be clever"; use what the model provider recommends, and add validation around it if you need additional guarantees. And when upgrading to a new model version, verify that tool-call formats haven't changed.