Tokens: How Agents See Text

What a token really is

Language models don't read text the way humans do. They don't parse character-by-character or word-by-word. Instead, a tokenizer -- a piece of software that runs before the model ever sees the text -- splits your input into chunks called tokens. Each token becomes a single numerical ID that the model processes.

The key insight: a token is neither a word nor a character. The word "beautiful" might be one token in some models, but in others, it splits into "beau" + "tiful" or "be" + "autiful". This variance comes from how the tokenizer was trained, and it has real consequences for token counts, costs, and where truncation happens.

How subword tokenization works

Most modern language models use subword tokenization, where a tokenizer learns a vocabulary of common character sequences (subwords) rather than treating every unique word as atomic. The most common algorithm is Byte Pair Encoding (BPE), which builds a vocabulary by repeatedly merging the most frequently co-occurring pairs of bytes or characters in a training corpus.

Here's the intuition: if the tokenizer was trained on English text, it learns that the sequence "ing" is very common (appearing, working, talking, etc.). So rather than tokenize "talking" as individual characters T-A-L-K-I-N-G (seven tokens), it encodes it as T-A-L-K + ING (four tokens). Even more common sequences like "the" might be a single token.

This works beautifully for common words and languages in the training data, but breaks down for rare words and languages the tokenizer never saw. A word like "tokenization" from a specialized domain might tokenize into many more pieces than an equally long common word. Non-English languages often tokenize less efficiently because their common sequences didn't appear as often in the training corpus.

Tokenization efficiency across languages and domains table

This table shows illustrative estimates of how many tokens the same semantic information requires across different languages and contexts. All entries represent roughly the same meaning: "What is the current price of Bitcoin?"

| Content type | Text | Token count (illustrative) | Tokens per word | Notes | |---------|---------|---------|---------|---------| | English | "What is the current price of Bitcoin?" | 7 tokens | 0.88 | Well-tokenized common domain | | Chinese | "比特币的当前价格是多少?" | 12 tokens | 1.5 | Less frequent in training corpus | | Japanese | "ビットコインの現在の価格は何ですか?" | 14 tokens | 2.0 | Character-based language | | Code (Python) | get_btc_price() | 5 tokens | 1.7 | Identifier names + special chars | | JSON API response | {"symbol":"BTC","price":42500.50} | 16 tokens | 2.7 | Structure overhead (brackets, colons) |

The same query requires 7 tokens in English but 14 tokens in Japanese, even though the information content is identical. For agents, this means non-English agents burn token budget faster, and code/JSON parsing tools are less efficient than they appear.

The impact: why token counts vary

Consider the sentence "I am using Python for NLP tasks." Let's see how tokenization varies:

# Using a simplified example of how three different tokenizers
# might handle the same text

text = "I am using Python for NLP tasks."

# Tokenizer A (character-level, inefficient baseline)
# i, space, a, m, space, u, s, i, n, g, ...
# Result: ~50 tokens

# Tokenizer B (word-based, uncommon)
# "I", "am", "using", "Python", "for", "NLP", "tasks", "."
# Result: 8 tokens

# Tokenizer C (BPE/subword, production language models)
# "I", "am", "us", "ing", "Python", "for", "NL", "P", "tasks", "."
# Result: 10 tokens (varies by exact vocabulary)

None of these are wrong; they're just different design choices. For agent design, the consequence is clear: you can't reliably predict token counts without knowing which model and tokenizer you're using. A tool result that takes 200 tokens in Claude Opus might take 280 in GPT-4, and 400 in a smaller model.

Estimating tokens in practice

A useful heuristic for English text: one token ≈ 4 characters, or roughly 3/4 of a word. So 100 words ≈ 130-140 tokens. This breaks down fast for code (which tokenizes inefficiently, often 1 token per character or fewer), non-English text (usually worse than English), and specialized vocabulary (much worse).

For real accuracy, you need to count. Here's a Python example using tiktoken (OpenAI's tokenizer, freely available):

import tiktoken

def estimate_tokens(text, model="gpt-3.5-turbo"):
    """Count actual tokens for a given text and model."""
    encoding = tiktoken.encoding_for_model(model)
    tokens = encoding.encode(text)
    return len(tokens)

# Example: comparing the same text across models
example_text = """
An AI agent is a system that perceives its environment through sensors
and acts upon that environment through actuators, guided by goals and
reasoning. Agents differ from chatbots in that they can take autonomous
action without waiting for user input between steps.
"""

gpt_count = estimate_tokens(example_text, "gpt-3.5-turbo")
gpt4_count = estimate_tokens(example_text, "gpt-4")
print(f"GPT-3.5-turbo: {gpt_count} tokens")
print(f"GPT-4: {gpt4_count} tokens")

For Anthropic models, you can use the official Python SDK's token counter:

import anthropic

def count_anthropic_tokens(text):
    """Count tokens for Claude models."""
    client = anthropic.Anthropic()
    response = client.messages.count_tokens(
        model="claude-3-5-sonnet-20241022",
        messages=[{"role": "user", "content": text}]
    )
    return response.input_tokens

example = "The quick brown fox jumps over the lazy dog."
token_count = count_anthropic_tokens(example)
print(f"This text uses {token_count} tokens in Claude")

Both approaches give you the real count for your specific model, which is essential when building agents that need to track context budget.

Why agents care about tokens specifically

In a single chatbot turn, you might not care that your prompt is 500 tokens. But in an agent loop that runs 20 turns, token counts matter constantly:

• Cost: most models charge per token. A tool result that takes 5,000 tokens instead of 500 because it wasn't summarized can cost 10x as much.
• Truncation decisions: when context fills up, the model or framework has to decide what to truncate. If you don't count tokens accurately, you can't predict where that truncation will happen.
• Speed: longer context means longer latency. Knowing token counts lets you optimize for speed, not just cost.
• Memory and long-term loops: agents that run continuously (handling multiple user requests in sequence, or taking many steps on one task) need to manage a token budget across time, exactly like a code project manages a memory budget.

Languages and special content tokenize differently

English tokenizes relatively efficiently. But the same principles mean:

• Code: often tokenizes poorly because special characters, indentation, and unusual identifier names aren't in the tokenizer's subword vocabulary. A 50-line Python script can easily be 3,000-4,000 tokens.
• JSON and structured data: similar problem to code; the structure overhead (quotes, colons, brackets) adds tokens without adding semantic content.
• Non-English languages: typically worse than English. A Chinese or Arabic sentence that conveys the same information as an English sentence often tokenizes to more tokens, because the training corpus for the tokenizer had less representation.
• Numbers and rare tokens: sequences of digits tokenize unpredictably. "2024" might be one token or four, depending on the tokenizer.

Real-world case study: a multilingual agent's token surprise

A travel booking agent was built to handle queries in English, Spanish, and Mandarin Chinese. The team counted tokens for the English version: system prompt + tool definitions + user query averaged 2,500 tokens, well within the 100k context window.

When they deployed in other languages, the token counts shocked them:

• English: 2,500 tokens
• Spanish: 3,200 tokens (28% overhead due to more verbose grammar)
• Mandarin Chinese: 4,100 tokens (64% overhead, far fewer tokens per character)

With 10 turns of agent reasoning + tool results, the Mandarin Chinese version hit context limits 2 turns earlier than English, and they had to truncate context or use shorter tool descriptions. The fix: redesign tool descriptions to be more concise in each language, or use a model with a larger context window for non-English agents.

Here's a concrete comparison:

texts = {
    "English sentence": "The quick brown fox jumps over the lazy dog.",
    "Python code": """def fibonacci(n):
    if n <= 1: return n
    return fibonacci(n-1) + fibonacci(n-2)""",
    "JSON structure": '{"user": "alice", "role": "admin", "active": true}',
}

for label, text in texts.items():
    tokens = estimate_tokens(text, "gpt-4")
    # English: ~9 tokens
    # Python: ~24 tokens (code is less efficient)
    # JSON: ~18 tokens (structure adds overhead)
    print(f"{label}: {tokens} tokens, {len(text)} chars, "
          f"efficiency: {len(text)/tokens:.1f} chars/token")

This difference matters in agent design: a web scraper that returns raw HTML (lots of tags, inefficient tokenization) will burn budget faster than one that extracts structured data, even if they contain the same information.

Token-efficient tool design patterns

Production agents should design tools to return compact representations. Here are three strategies:

# Strategy 1: Tiered response (summary by default, detail on request)
class SearchResult:
    def __init__(self, title: str, summary: str, full_text: str):
        self.title = title
        self.summary = summary
        self.full_text = full_text

    def __str__(self):
        # When inserted into context, return summary (~100 tokens)
        return f"{self.title}: {self.summary}"

    def get_full(self):
        # Only called if agent explicitly asks for it
        return f"{self.title}\n\n{self.full_text}"

# Strategy 2: Structured extraction (parse before returning)
def fetch_company_info(company_name: str) -> dict:
    """
    Scrape company website, but return structured facts, not raw HTML.
    """
    html = scrape_website(company_name)

    # Extract key facts using NLP/parsing
    facts = {
        "name": company_name,
        "founded": extract_founded_year(html),
        "headquarters": extract_location(html),
        "employees": extract_headcount(html),
        "revenue_status": extract_revenue_range(html),  # e.g., "$1M-10M"
    }

    return facts  # ~80 tokens for the JSON, vs. 8,000 for raw HTML

# Strategy 3: Smart filtering (return only relevant results)
def search_documents(query: str, max_tokens: int = 500) -> str:
    """
    Search documents, but limit results to fit token budget.
    """
    results = vector_search(query, top_k=20)

    output = []
    token_count = 0

    for result in results:
        result_tokens = estimate_tokens(result.text)
        if token_count + result_tokens > max_tokens:
            break  # Stop adding; we've hit the budget
        output.append(result)
        token_count += result_tokens

    return format_results(output)

Tools designed this way scale better: the agent doesn't need to page through results or re-request with different parameters because size limits were hit midway.

Estimating tokens for complex structures

For tools that return structured data (JSON, tables, etc.), token estimates are surprisingly non-linear. A 10-item list of objects isn't just 2x the tokens of a 5-item list; the structure overhead (brackets, commas, key names) is amortized less efficiently as lists grow. Here's a practical rule:

# For lists/tables, actual tokens ≈ base_overhead + (num_items × per_item_cost)
# base_overhead: ~50 tokens (opening brackets, structure)
# per_item_cost: ~30 tokens for a simple row/object

def estimate_list_tokens(num_items: int, per_item_avg: int = 30) -> int:
    base_overhead = 50
    return base_overhead + (num_items * per_item_avg)

# Example:
# 10 items: 50 + (10 * 30) = 350 tokens
# 100 items: 50 + (100 * 30) = 3,050 tokens

# BUT: raw HTML for the same data:
# 10 items: ~800 tokens (lots of tags)
# 100 items: ~8,000 tokens (tags accumulate)

# Structured is 2–3x more efficient

This is why agents designed with structured tools are more token-efficient than those fetching raw HTML, even if the semantic content is identical.

A real worked example: agent design with token awareness

Say you're building a research agent that searches the web, reads articles, and synthesizes findings. You know:

• System prompt + tool definitions: 1,500 tokens (fixed)
• Each search query: 15 tokens
• Each search result (returned as a snippet): 200 tokens
• Raw HTML page dump: 8,000 tokens
• Structured extracted data from a page: 400 tokens

With a 100,000-token context window:

• Bad design: fetch raw HTML on every page load. After 10 pages, you've used 80,000 tokens (1,500 + 150 search queries + 80,000 HTML) and have room for only 2-3 turns of reasoning before hitting the limit.
• Good design: parse pages server-side, extract structured facts (vendor, price, key specs), return that. Same 10 pages now cost 1,500 + 150 + 4,000 tokens, leaving 94,000 for many more reasoning steps and refines searches.

The second agent handles the same task, queries the same sources, but completes in half the tokens because it counted tokens, understood tokenization inefficiency, and designed its tool outputs accordingly.

Edge case: tokenization discontinuities

Some tokenizers have discontinuities where a small change in input causes a large jump in token count. For example:

# These look similar semantically, but tokenize very differently:

# Case 1: Common URL
text1 = "https://www.example.com/api/users"
# Tokenizes efficiently: URL patterns are common
# ~6 tokens

# Case 2: Slightly malformed or custom URL
text2 = "https://www-staging.example-internal.corp/v2/api/users"
# Contains unusual subdomains and hyphenated domain parts
# ~20 tokens (3x overhead!)

# Case 3: Numeric ID vs. random string
text3 = "user_12345"
# Common pattern (numeric IDs are frequent)
# ~2 tokens

# Case 4: Random slug
text4 = "user_xkqp9"
# Uncommon character sequence
# ~4 tokens (2x overhead)

This matters in agent design: if your tool descriptions or examples use atypical patterns (custom domain names, unusual formats), they'll tokenize less efficiently than you expect. Use realistic, common patterns in your examples, and you'll avoid this surprise.

Edge case: token counting at scale

For a single agent run, miscounting tokens by 10-20% is annoying but usually harmless. But at scale, in a platform serving thousands of agents, small errors compound. If you estimate 100 tokens per agent run and undercount by 20%, you've just budgeted 10% less capacity than you actually need -- and at 100,000 runs/day, that's 200,000 wasted tokens per day across your entire platform.

Production systems should:

• Count tokens once per unique input pattern (not per run)
• Add a 15-20% safety margin to all estimates
• Monitor actual vs. estimated token counts in production and adjust budgets quarterly
• Use the exact tokenizer from each model provider (not approximations)

A small investment in token counting infrastructure (a cache of counted examples, periodic validation) saves surprises.

Common mistake

Assuming your token estimates transfer across models. A prompt that is 1,200 tokens in GPT-4 might be 1,100 tokens in Claude or 1,400 tokens in Gemini, even though it's identical text. When switching models, recount your tokens. Similarly, don't assume "tokens ≈ words" works for code or structured data; count the real thing. The difference between planning for 500 tokens and discovering you actually needed 3,000 mid-agent-run is the difference between a predictable cost and a surprising bill. And at scale, small counting errors compound into large budget misses.