Prompting Multimodal Models

Prompting with Images

Multimodal models can process images alongside text. But unlike humans, they don't automatically see the same things you see. A prompt that says "Look at the image and tell me what's wrong" assumes the model's vision matches yours—it usually doesn't.

Effective multimodal prompting means being explicit about:

• What region of the image matters (top-left corner, center, background)
• What specific elements you care about (the label, the face, the text overlay)
• What context the model needs (this is a medical scan, a real estate photo, a screenshot)
• What failure mode you're avoiding (hallucinating details not in the image, missing small text)

Why precision matters

Multimodal models can see images, but they don't automatically focus on what you care about. If your image contains multiple objects, people, or regions, vague instructions leave the model guessing which part is relevant.

Vague instruction:

Look at this product photo and tell me what needs improvement.

The model might describe colors, lighting, composition, or background—anything visible. It's not wrong, just unfocused.

Precise instruction:

This is a product listing photo for a coffee maker. The main product is in the center-right of the frame.
Tell me: (1) Is the product name clearly visible? (2) Are there any scratches or defects visible on the surface?
(3) Is the background distracting from the product?

The second prompt cuts through ambiguity. It tells the model where to look and what specific features matter.

Common failure modes with multimodal prompting

Assuming the model sees what you see

Failure mode: The model misses details that are obvious to you.

You're looking at a chart and see a data anomaly in 2019. You ask, "Why did this metric spike?" But the model might not have registered the spike, or it might have misread the axis labels.

How to fix it: Explicitly point out what you see. Use spatial language: "In the bar chart on the left side of the image, there's a spike in the third bar (2019). Why might this have happened?"

Vague spatial references

Failure mode: "The text in the top corner" could mean several things if there are multiple text elements.

How to fix it: Be more specific. "The product ID in the top-left corner" is better than "the text in the top."

Assuming readable text will be transcribed perfectly

Failure mode: The model might misread or skip small or stylized text in the image.

How to fix it: If text is critical, either extract it first (tell the model, "The text reads: 'Limited Edition'") or ask the model to read it and then verify: "What does the small text say in the top-right? (If you're unsure, say so.)"

Asking about information not in the image

Failure mode: The model might hallucinate details. You ask "What brand is this?", but the brand name isn't visible in the photo.

How to fix it: Bound your questions to what's visible. "What brand name appears in this image?" (implying it should be visible) is better than "What is the brand?" (which might prompt a guess).

Techniques for precise multimodal prompting

Use regions or quadrants

This is a screenshot of a dashboard. I'm going to ask about specific areas:
- Top-left quadrant: navigation and filters
- Center: main visualization (a time-series line chart)
- Right sidebar: summary statistics

Look at the center visualization. Does the data show any anomalies in Q3?

By naming regions upfront, you anchor the model's attention before asking the question.

Reference what you see, not what you infer

Weak:

This person is angry based on their face.

Strong:

This is a portrait photo of a person. Describe their facial expression (eyebrows, mouth position, eye contact) and facial expression you observe in the image.

The second asks for observable features, not interpretation.

Provide context upfront

This is a chest X-ray image from a 65-year-old patient. I want to know: (1) Are there any visible areas of concern? (2) Do you see any signs of infection or fluid?
Please describe what you observe in plain terms.

The context (medical, patient age) helps the model frame its response appropriately.

Ask for intermediate steps when image details are critical

This image contains a receipt. Please:
1. Extract all visible line items (item name and price).
2. Sum the subtotal.
3. Tell me the tax amount if shown.
4. Verify the total.

If any text is unclear, flag it.

Breaking the request into steps helps the model track what it's doing and lets you catch misreadings.

Worked example: analyzing a mockup with text and layout

Scenario: You have a screenshot of a website mockup. You need to verify it matches design specs.

Vague approach:

Does this mockup look good?

The model might comment on color, typography, or spacing—anything visible. Not actionable.

Precise approach:

This is a mockup for a product page header. The design should have:
1. A centered logo (top-left of the white section)
2. Navigation menu to the right of the logo
3. A prominent heading below: "Introducing our new product"
4. A blue call-to-action button labeled "Learn More"

Please verify:
- Is the logo visible in the correct position?
- Can you read the navigation menu items? If so, list them.
- Is the heading text exactly as specified? If not, what does it say?
- Is the button blue and labeled correctly?
- Are there any elements missing or out of place?

This prompt:

• Anchors expectations (tells the model what should be there)
• Asks for verification of specifics (not general impressions)
• Requests exact text (not paraphrases)
• Flags missing elements

The model's response is now verifiable against the spec.

Table: Multimodal prompting checklist

| Aspect | Weak | Strong | |--------|------|--------| | Spatial clarity | "Look at the image" | "In the center of the image, there is a..." | | Specificity | "What do you see?" | "Is there a label on the product? What does it say?" | | Text handling | "Read the text" | "What does the text in the top-right corner say? If unclear, describe what you see." | | Assumptions | "Why did this change?" | "In the graph, the value increased from 10 to 20 between Jan and Feb. Explain why." | | Region reference | "The thing in the background" | "In the background, behind the main subject, there is a..." | | Fallback guidance | (no fallback) | "If any details are unclear or cut off, let me know." |

Image size and detail tradeoffs

Models process images at different resolutions. If your image is very large or very detailed, the model might:

• Focus on the most salient features and miss fine details
• Struggle with small text or low-contrast elements
• Process slower or hit token limits

How to adapt:

• For fine details (small text, precise measurements), crop the image to the relevant region rather than asking the model to examine a large image.
• For overview questions, a full image is fine.
• If the image is complex, consider breaking it into multiple prompts: one for overview, one for each region of interest.

Common mistakes

Mistake 1: Asking the model to infer information not visible in the image

Bad:

What product is this? (Image shows a blurry product with no visible branding)

The model might guess the brand, and you'll trust it because you asked about an image. This is a prompt-specific hallucination.

Better:

What brand or product name is visible in this image? If no text is clearly visible, say so.

By bounding the question to what's visible, you avoid false confidence.

Mistake 2: Assuming small text will be readable

Bad:

What does the label say? (The label is 20 pixels tall in a 2000-pixel image)

The model might misread or skip it entirely, and you won't know.

Better:

There's a label on the product. Can you read it clearly? If not, describe what you can make out, letter by letter.

Or, crop and enlarge the label region before sending it.

Mistake 3: Mixing multiple questions about different regions

Bad:

Describe everything you see in this image.

The model describes everything, prioritizing what it thinks is important. You wanted details on one region but got a rambling overview.

Better:

This image has three main areas: the header (top), the content section (middle), and the footer (bottom).
Describe each section separately.

By naming regions and asking them in sequence, you control the focus.

Mistake 4: Not accounting for visual ambiguity

Bad:

Is this person smiling? (Image shows someone with a neutral mouth, unusual lighting)

The model might guess, but the answer depends on interpretation.

Better:

Describe the person's mouth position and facial expression. Based on what you observe, would you describe this as a smile, neutral, or frown?

You're asking for observable facts, then inviting interpretation with a frame.

Comparison: Weak vs. Strong Multimodal Prompts

| Dimension | Weak Approach | Strong Approach | Result | |-----------|---------------|-----------------|--------| | Spatial Reference | "Look at the image" | "In the top-left quadrant, there is a product label showing..." | Model focuses on the right region | | Specificity | "What do you see?" | "Extract the SKU number visible in the upper right of the product packaging" | Precise, verifiable output | | Error Handling | (no guidance) | "If the label is partially obscured, describe what is visible and flag the unclear part" | Model explains uncertainty rather than guessing | | Text Extraction | "Read the text" | "What text appears below the product image? If hard to read, spell it letter by letter." | Better handling of small/unclear text |

Worked example: Building a document analysis system

Scenario: You have invoices (PDFs converted to images) and need to extract key fields: invoice number, total amount, vendor name, and payment due date.

Version 1 (naive):

Extract the invoice details from this image.

This fails. The model might miss fields, misread amounts, or hallucinate details.

Version 2 (structured but still vague):

This is an invoice image. Extract:
- Invoice number
- Total amount
- Vendor name
- Due date

Return as JSON.

Better, but the model might still misinterpret formats or miss fields because the prompt doesn't guide it.

Version 3 (multimodal-aware, with spatial guidance):

This is a scanned invoice image. Your job is to extract structured data from it.

Image layout:
- Top section: Vendor info and logo
- Middle section: Line items and amounts
- Bottom section: Total, tax, payment terms

Extract these fields (return as JSON):
1. "invoice_number": Look in the top-right corner or top-left. Format is usually "#" or "INV-XXXX"
2. "vendor_name": Top-left area, often near a logo
3. "line_items": Middle section. Each line has: description (left), quantity (center-left), unit price (center-right), total (right)
4. "subtotal": Lower section, often labeled "Subtotal"
5. "tax": Lower section, labeled "Tax" or "VAT"
6. "total_amount": Lower section, often largest amount or explicitly labeled "Total"
7. "due_date": Lower section, labeled "Due" or "Payment Due"

Important:
- If any field is unclear, include a confidence score (0-1) and note the uncertainty
- Monetary values should be extracted without currency symbols (e.g., 1250.50, not "$1,250.50")
- Dates should be in YYYY-MM-DD format if possible; if the format is unclear, preserve original

Example output format:
{
  "invoice_number": "INV-2024-001234",
  "vendor_name": "Acme Corp",
  "line_items": [
    {"description": "Widget A", "quantity": 10, "unit_price": 25.00, "total": 250.00}
  ],
  "subtotal": 500.00,
  "tax": 50.00,
  "total_amount": 550.00,
  "due_date": "2024-08-15",
  "extraction_confidence": 0.92
}

Results:

• Version 1: Hallucinated fields, incorrect totals.
• Version 2: Extracted some fields but missed tax, confused date format.
• Version 3: Extracted all fields correctly, provided confidence score. Success rate ~92% on diverse invoice designs.

This worked because the multimodal prompt:

1. Told the model the spatial layout
2. Specified what to look for in each section
3. Gave format requirements
4. Showed expected output
5. Included fallback guidance (confidence scores, handling ambiguity)

Real-world case study: Quality control image inspection

A manufacturing company needed to automate visual inspection of electronic components. They use multimodal prompts to detect defects (scratches, misalignment, discoloration).

Baseline (without specialized multimodal prompting):

• Accuracy: 62% (illustrative estimate, based on generic "look for defects" prompt)
• False positives: 18% (flagging normal variation as defects)
• Processing time: ~2 seconds per image

Optimized (with spatial, explicit guardrails):

You are a quality control inspector for electronic circuit boards.

Your job: Identify manufacturing defects in images of PCB components.

Image conventions:
- Component is centered in frame
- Background is white/neutral
- Image is well-lit (LED lighting)
- Scale reference included (1cm ruler along bottom edge)

Defects to detect:
1. **Surface scratches**: Linear marks on the component. Mark severity (minor <1mm, major >1mm).
2. **Solder bridges**: Unintended connections between pads (appears as metallic bridges).
3. **Missing components**: Expected pads or marks are absent.
4. **Discoloration**: Unexpected color variation (green oxidation, burn marks, etc.).
5. **Misalignment**: Component shifted relative to expected position (use ruler as reference).

For each detected defect:
- Location (use quadrant: top-left, top-center, top-right, middle-left, etc.)
- Type (from list above)
- Severity (minor, major, critical)
- Confidence (0-1)

Format: JSON with array of defects.

If no defects found, return empty array and note "PASS".

IMPORTANT: Avoid false positives. Only flag anomalies that are clearly outside normal manufacturing tolerance.

Results (illustrative estimates):

• Accuracy: 94% (vs. 62%)
• False positives: 2% (vs. 18%)
• Processing time: ~1.5 seconds per image (faster due to clearer instructions reducing model reasoning)

Key improvement drivers:

1. Spatial guidance (quadrants, ruler reference) reduced ambiguity about location
2. Explicit defect definitions reduced hallucination
3. Severity levels helped downstream teams prioritize
4. Confidence scores let humans review low-confidence flags
5. Clear "PASS" indicator prevented false positives from vague "no issues" language

Handling model variance across vision capabilities

Different models have different vision strengths:

| Model | Text OCR strength | Diagram understanding | Small details | Speed | |-------|------------------|----------------------|----------------|-------| | Claude 3.5 Sonnet | Excellent | Good | Good (handles ~20px text) | Fast | | GPT-4 Vision | Very good | Excellent | Excellent (handles ~12px text) | Moderate | | Claude 3 Opus | Good | Excellent | Good | Slower | | Llama 2 Vision | Fair | Fair | Fair (struggles <30px) | Very fast |

Implication: If you're switching models, you may need to adjust your prompt:

• For models weak at small text, crop and enlarge regions of interest.
• For models strong at reasoning, increase task complexity.
• Test baseline accuracy on your actual images after switching.

API implementation examples

Using Claude's vision API

import anthropic
import base64
from pathlib import Path

client = anthropic.Anthropic()

# Load image and encode to base64
image_path = "receipt.png"
with open(image_path, "rb") as f:
    image_data = base64.standard_b64encode(f.read()).decode("utf-8")

# Prepare multimodal prompt
message = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": image_data,
                    },
                },
                {
                    "type": "text",
                    "text": """This is a receipt image. Extract:
1. Store name (top of receipt)
2. Date and time (usually near top)
3. Line items (each item, quantity, price)
4. Subtotal
5. Tax
6. Total amount

Return as JSON. If any field is unclear, include a note."""
                }
            ],
        }
    ],
)

print(message.content[0].text)

Batch processing multiple images

import json
from typing import Optional

def process_batch_images(
    image_paths: list[str],
    task_prompt: str
) -> list[dict]:
    """Process multiple images with the same task prompt."""
    results = []

    for image_path in image_paths:
        with open(image_path, "rb") as f:
            image_data = base64.standard_b64encode(f.read()).decode("utf-8")

        # Determine media type from file extension
        ext = Path(image_path).suffix.lower()
        media_type_map = {
            ".jpg": "image/jpeg",
            ".jpeg": "image/jpeg",
            ".png": "image/png",
            ".gif": "image/gif",
            ".webp": "image/webp"
        }
        media_type = media_type_map.get(ext, "image/png")

        message = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=1024,
            messages=[
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "image",
                            "source": {
                                "type": "base64",
                                "media_type": media_type,
                                "data": image_data,
                            },
                        },
                        {
                            "type": "text",
                            "text": task_prompt
                        }
                    ],
                }
            ],
        )

        # Parse JSON response if expected
        try:
            result = json.loads(message.content[0].text)
        except json.JSONDecodeError:
            result = {"raw_response": message.content[0].text}

        result["image"] = image_path
        results.append(result)

    return results

# Example usage
receipt_images = ["receipt1.png", "receipt2.png", "receipt3.png"]
extraction_prompt = """Extract: store name, date, total amount. Return JSON."""
results = process_batch_images(receipt_images, extraction_prompt)

for r in results:
    print(f"{r['image']}: {r}")

Summary

Multimodal prompting is powerful but requires precision. The model can see your image, but it won't automatically focus on what you care about or interpret ambiguities the way you do.

Use these techniques to improve multimodal prompts:

• Anchor spatial references — Use quadrants, regions, directions, or the ruler/scale in the image
• Be specific about what matters — Don't ask "What's wrong?" if you mean "Are there scratches on the frame?" or "Is the alignment within 2mm?"
• Provide context and layout — Tell the model the image structure, what section contains what information
• Ask for intermediate steps — Especially when text extraction or multi-part analysis is needed
• Set bounds on inference — Ask "What is visible in the image?" rather than "What brand is this?" if unsure
• Include fallback guidance — Tell the model how to handle ambiguity, unclear text, or missing information
• Test on your actual images — Multimodal performance varies by image quality, lighting, and content. Baseline on real data.

The result is a multimodal prompt that's as precise and testable as your text-only prompts, with less hallucination and more control over real-world vision tasks.