Multimodal Models in Agent Systems

Beyond text in, text out

A multimodal model can accept input beyond plain text -- images, PDFs, audio, sometimes video frames -- and reason about it alongside text in the same context window. For an agent, this means a tool result or a piece of retrieved context doesn't have to already be converted to text before the model can use it: a screenshot of a webpage, a scanned invoice, or a chart image can go directly into the model's context.

The distinction matters. A text-only agent receiving a tool result must rely on whatever text extraction happened before the model saw it: OCR for scanned documents (error-prone), HTML parsing for webpages (loses visual state), or data structure extraction for images (loses nuance). A multimodal agent can see the raw visual input and draw its own conclusions about what's important, what's highlighted, what's disabled or greyed out, and what the overall layout communicates.

This is not about "smarter" models -- it's about information fidelity. OCR of a scanned check might turn "$1,234.56" into "$1,23456" or fail entirely on handwriting. A multimodal model looking at the image directly has no such conversion error. The model still might misread a number, but the error is at the reasoning level, not the transcription level.

Where this actually changes agent design

Multimodal input makes a meaningful difference when the information you need is conveyed visually or acoustically, not just as transcribed text. These are not edge cases; they're common agent scenarios:

Browser-using agents

A screenshot-aware agent can interact with a web application by seeing the actual rendered state of the page. If a button is greyed out, the screenshot shows that -- the HTML alone might not mark it as disabled. If a modal popup appeared, covering the main content, the screenshot catches that immediately. An OCR-driven text extraction might miss the modal entirely or confuse it with page content.

A worked example: an agent accessing a form with conditional fields. If answering "Do you own a car?" with "No" hides the "vehicle mileage" field, a text-only agent might request that field anyway, getting an error or empty response. A screenshot-aware agent sees the hidden field and skips it.

Document processing agents

A classic example: expense reports. A scanned receipt or invoice contains:

• Printed text (vendor name, date, total)
• Visual layout (table structure, receipt format)
• Stamps or signatures (approval indicators)
• Handwritten annotations (notes, corrections)

A text-only agent needs all of this converted to clean data first. OCR might fail on faint print or handwriting. Layout extraction (identifying which numbers are line items vs. totals) requires separate logic. A multimodal agent reads the image directly: it sees the vendor's letterhead, the total circled in red, the initials of the approver, and the note "reimbursed 2/28" scrawled at the bottom. It extracts meaning from the visual context, not just character sequences.

Data analysis and chart understanding

A chart is inherently visual. You could provide a model with a data table and ask it to analyze the trend, but a time-series chart shows patterns (seasonality, inflection points, outliers) in a glance. When an agent's job is "identify what's unusual about this dataset," a chart image often provides better signal than the underlying numbers, because the agent can spot visual anomalies (a sudden spike, a plateau, a discontinuity) without needing you to explain what to look for.

Voice and audio agents

Audio carries information beyond the words: tone of voice, confidence or hesitation in speech, background noise, interruptions, overlapping voices in a conversation. If an agent's job is to categorize a customer support call, the transcript alone loses the emotional context that a frustrated customer's tone conveys. The agent cannot distinguish between "that's fine" said with genuine acceptance and "that's fine" said with barely-suppressed irritation.

The cost and latency tradeoffs

Multimodal inputs consume more tokens than their text equivalent. An image that contains 500 words of text doesn't tokenize to 500-700 tokens; it typically costs 500-2,000 tokens, depending on image resolution and the model's vision encoder. A high-resolution screenshot (2560x1440) can cost 3,000-4,000 tokens. A short audio clip might cost more tokens than the equivalent transcript.

Multimodal calls are often slower. Processing an image adds vision encoding latency on top of model inference. A text-only call might take 800ms; the same model processing a large image might take 2-3 seconds. For agents that make many sequential requests, this compounds.

Cost and latency comparison for common multimodal inputs (illustrative estimates based on Claude 3.5 Sonnet pricing as of mid-2026):

| Input type | Approx. token cost | Approx. latency | Best for agents | |---|---|---|---| | Text snippet (100 words) | 150-200 tokens | 0.3-0.8s | All text-based reasoning | | Low-res image (512x512) | 500-700 tokens | 1.2-1.8s | Thumbnails, small screenshots | | Standard screenshot (1920x1080) | 1,500-2,000 tokens | 1.8-2.5s | Web interaction, UI verification | | High-res image (2560x1440) | 3,000-4,000 tokens | 2.5-3.5s | Fine detail (charts, text legibility) | | PDF page (as image) | 2,000-3,500 tokens | 2.0-3.0s | Document layout verification | | Audio clip (10 seconds) | 1,200-2,000 tokens | 2-4s | Tone/emotion detection, speaker ID | | Audio transcript (equivalent to above) | 150-300 tokens | 0.3-0.8s | Content extraction only |

The practical implication: reserve multimodal calls for the specific steps in an agent loop that genuinely need visual or audio understanding. Most steps don't. A research agent that retrieves an article should do so as text, not as an image of each webpage. But if the research includes analyzing a competitor's visual product (a screenshot of their dashboard, a design mockup), that step is worth the cost.

Here's a pattern for cost-efficient multimodal agents:

def process_document_step_by_step(document_image, document_text):
    """
    First classify the document visually, then extract details from text.
    Only the classification step uses the expensive image input.
    """
    client = anthropic.Anthropic()

    # Step 1: Classify the document type using vision (needed for context)
    classification = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=100,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": document_image  # Only here
                    },
                },
                {
                    "type": "text",
                    "text": "What type of document is this? (Receipt, Invoice, Claim form, etc.)"
                }
            ],
        }],
    )
    doc_type = classification.content[0].text

    # Step 2: Extract specifics using text-only (cheaper and faster)
    # The model already knows the document type from step 1
    if doc_type == "Invoice":
        extraction = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=200,
            messages=[{
                "role": "user",
                "content": f"""This is an {doc_type}.
Extract: vendor name, invoice number, date, and total amount.

Text extracted from the document:
{document_text}"""
            }],
        )
    # ... handle other doc types similarly

    return {
        "type": doc_type,
        "details": extraction.content[0].text
    }

This pattern uses vision only where it adds value: document classification (where visual layout matters). Details are extracted from text, which is cheaper and doesn't lose information once you know what you're looking for.

Multi-step efficiency: reuse extracted text

Once you've extracted text from an image or audio, reuse it for subsequent steps. An agent analyzing a scanned contract might:

1. Step 1 (multimodal): Read the image to confirm it's a valid contract and identify the signing date and parties (requires layout understanding).
2. Step 2 (text-only): Extract all liability clauses using the OCR'd text (text extraction is sufficient).
3. Step 3 (text-only): Compare extracted clauses to a template (pure text analysis).

After step 1, the image is no longer needed. Steps 2 and 3 use the extracted text, saving tokens and latency for the remaining 80% of the work.

def analyze_contract_efficiently(contract_image: bytes, contract_ocr_text: str):
    """
    Analyze a contract, using vision only where necessary.
    """
    client = anthropic.Anthropic()

    # Step 1: Vision step - validate and extract dates (requires layout understanding)
    validation = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=200,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": contract_image
                    },
                },
                {
                    "type": "text",
                    "text": "Is this a valid contract? Extract the signing date and parties. Look at actual signatures."
                }
            ],
        }],
    )

    # Step 2 & 3: Text-only - analyze terms using OCR'd text
    analysis = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=500,
        messages=[{
            "role": "user",
            "content": f"""Document validation: {validation.content[0].text}

Now analyze liability clauses in this contract text:
{contract_ocr_text}

Flag any clauses that expose the company to unlimited liability."""
        }],
    )

    return {
        "validation": validation.content[0].text,
        "liability_analysis": analysis.content[0].text
    }

This approach uses vision once, for the task where it genuinely adds value. All subsequent analysis works from text, which is efficient and usually sufficient.

Edge cases and failure modes in multimodal agents

Hallucination with images: A multimodal model can confidently describe details that aren't in an image. If a receipt is blurry, the model might "read" a number that isn't actually legible. This is more dangerous than text hallucination because the visual input feels authoritative. Always include a confidence check: "If you cannot read a value clearly, say 'unreadable' rather than guessing."

Image quality variation: An agent that works reliably on clear, well-lit screenshots might fail silently on:

• Rotated or skewed images (from a phone photo of a document)
• Low-contrast or faded scans
• Partially obscured images (items out of frame, text covered by shadows)

This requires preprocessing: auto-rotate images, enhance contrast, or resizing before sending to the model. Sometimes it's cheaper to use a specialized OCR service (which fails explicitly on bad images) rather than a multimodal LLM (which fails silently by hallucinating).

Token cost explosion with batch operations: If an agent processes 1,000 images, the multimodal cost compounds massively. A naive approach embedding all 1,000 high-resolution images costs 2-4 million tokens. A smarter approach uses a two-step process: (1) downscale and embed in bulk with a fast service, (2) use multimodal LLM only on the top-K results that pass filtering. This can reduce multimodal calls from 1,000 to 10-50.

Audio timing sensitivity: In conversational or real-time scenarios, audio processing is significantly slower than text processing. A customer service agent that can ingest text chat handles 10x more throughput than one requiring audio processing. Use audio only when tone/emotion context is genuinely necessary.

Here's a practical check before adding multimodal to an agent step:

def should_use_multimodal(task_info: dict) -> bool:
    """Decide whether multimodal input is cost-justified."""

    task_type = task_info.get("type")
    data_volume = task_info.get("num_items", 1)
    token_budget = task_info.get("max_tokens_available", 10000)

    # Check 1: Is the information genuinely visual/audio only?
    visual_only = task_info.get("visual_only", False)
    if not visual_only:
        return False  # Text extraction sufficient

    # Check 2: Can we afford the token cost at scale?
    estimated_tokens_per_item = 2000  # midpoint estimate
    total_estimated_tokens = data_volume * estimated_tokens_per_item

    if total_estimated_tokens > token_budget:
        return False  # Too expensive; use text extraction instead

    # Check 3: Is there a faster alternative?
    text_extraction_available = task_info.get("text_extraction_available", False)
    if text_extraction_available:
        # Use text extraction first, multimodal only on outliers
        return False  # Do two-stage approach instead

    # If we pass all checks, multimodal is justified
    return True

# Example usage:
task = {
    "type": "document_classification",
    "num_items": 100,
    "visual_only": True,
    "text_extraction_available": True,
    "max_tokens_available": 200000
}
print(should_use_multimodal(task))  # False: use text extraction first

A worked example: full e-commerce receipt audit agent

An agent audits receipts for compliance with company policy. Here's how a multimodal-aware design differs from a naive one:

Naive approach: Load all 100 scanned receipts as high-resolution images, feed each one to the model with the question "is this compliant?" The model processes 100 images, each costing 3,000 tokens. Total cost: 300,000 tokens just for the image encoding, plus reasoning. Slow, expensive.

Multimodal-aware approach:

1. Preprocessing (before the agent loop): Batch OCR all receipts into text, using a fast, cheap OCR service or library. Cost: minimal.
2. Agent step 1: Use text-only analysis to classify each receipt (receipt vs. invoice vs. other). Cost: 100 tokens per receipt.
3. Agent step 2: For receipts that pass initial classification, use text extraction to pull vendor, date, amount, category. Cost: 200 tokens per receipt.
4. Agent step 3 (multimodal, only when needed): For receipts that fail category matching or have ambiguous amounts, load the image and do a detailed visual inspection. Cost: 2,000 tokens per ambiguous receipt (probably 5-10% of the batch).
5. Agent step 4 (text-only): Compare extracted data against policy rules. Cost: 100 tokens per receipt.

Total cost for multimodal approach: 100×100 + 100×200 + 7×2000 + 100×100 ≈ 30,000 tokens. This is 10x cheaper than the naive approach and actually more accurate because the model focuses multimodal attention where it's most needed -- on the edge cases, not the routine ones.

Case study: Document processing at a financial services firm

The problem: A mid-size fintech company processes 500 customer-submitted documents per day: bank statements, tax returns, identity proofs, and proof-of-address documents. Historically, they used a text OCR service followed by manual human review of ambiguous extractions. This worked but was slow and required 2-3 hours of daily human review to catch OCR errors.

Naive multimodal approach: Feed all 500 scanned documents directly to Claude's vision API to extract customer data and verify authenticity. Cost: 500 docs × 2000 tokens/doc = 1M tokens/day ≈ $15-20/day (with Claude 3.5 Sonnet pricing). Response time: 500 calls × 2-3s latency = 1,000-1,500 seconds if sequential, or ~5-10 seconds if parallelized.

Actual cost-optimized approach:

1. Batch OCR preprocessing (< $1/day): Use a dedicated OCR service (Google Vision or AWS Textract) on all 500 documents in parallel. Cost is ~$0.20/document but they get fast turnaround and can explicitly flag confidence scores. Output: text + confidence scores.
2. Filtering (0 API calls): Documents with OCR confidence > 95% are marked "approved." Documents with confidence 70-95% are flagged for step 3. Documents < 70% are routed directly to human review.
3. Selective multimodal verification (~100 API calls): Only the ambiguous middle group (usually 8-15% of docs) gets sent to Claude for visual verification. The model sees the OCR'd text and the image, and can confirm or correct. Cost: 100 docs × 2000 tokens = 200k tokens ≈ $3/day.
4. Human review (15 minutes instead of 2-3 hours): Humans now only review the few cases where OCR and multimodal both flagged uncertainty or contradicted each other.

Total daily cost: $1 (OCR) + $3 (multimodal verification) = $4/day vs. $15-20/day for naive approach, plus 95% reduction in human review time. The system is actually more accurate than pure OCR, because the multimodal step catches cases where OCR was confidently wrong.

Common mistake

Feeding a full-resolution screenshot or a lengthy audio clip into every single step of an agent loop "just in case" it's useful. Multimodal input should be requested by the specific step that needs it -- most iterations of a loop don't need to re-see the same image, they need the text conclusion the model already drew from it on a previous step. If you find yourself sending the same image to the model five times in a row, you're doing it wrong. Extract once, reuse the text.