Prompting for Structured Output (JSON and Beyond)

Language models are naturally fluent, conversational, and imprecise. When you need data in a specific format — JSON for a database, CSV for a spreadsheet, XML for a configuration system — you must actively guide the model toward that structure. This lesson covers the most reliable techniques: schemas, examples, API features, and validation.

The challenge is fundamental: language models predict text one token at a time. They are trained on human-written text, which is often informal, conversational, and full of tangents. When you ask for structured output, you are asking the model to do something it was not optimized for — to emit tokens in a very specific order, with exact punctuation and no flexibility. This is entirely possible, but only if you are explicit about what "structured" means and provide the model with clear guidance.

The Problem: Unstructured Output by Default

Consider a naive request:

Extract the customer name, email, and phone number from this message as JSON.

Customer message: "Hi, I'm Sarah Chen. You can reach me at [email protected] or 555-0142."

A model might respond with:

{
  "name": "Sarah Chen",
  "email": "[email protected]",
  "phone": "555-0142"
}

But it might also respond with:

The customer's information is:
- Name: Sarah Chen
- Email: [email protected]
- Phone: 555-0142

Or even mix partial JSON with prose. Without explicit constraints, you are asking for format consistency but not requiring it. The model will do its best to guess, and "best guess" in production systems leads to broken parsers and silent data loss.

Why Structured Output is Hard

Models generate tokens sequentially and greedily — always predicting the token most likely to appear next, given the context so far. A model trained on billions of words of human-written text learned that prose is the "default" output. JSON, CSV, and XML are structured, but they are a tiny fraction of training data compared to narrative text.

When you ask for structured output without explicit guidance, the model is making a guess about what you want. It might think:

• You want a formatted response but do not care if it is valid format.
• You want the key information formatted, but prose preambles are fine.
• You want mostly JSON but can mix in English explanations.

Real-world failure rates (illustrative estimates based on typical production systems):

| Scenario | Failure Rate | Typical Error | |----------|---|---| | Naive "Output as JSON" (no schema, no examples) | 20–40% | Preamble before JSON, malformed output, missing fields | | Explicit schema only | 5–15% | Type errors, optional fields missing | | Schema + 1–2 examples | 2–5% | Edge case handling, null handling | | Native JSON mode / function calling | <1% | Extremely rare; usually a model bug |

This is why adding explicit structure is so critical. You are not just asking; you are constraining. The difference between "output as JSON" and "output ONLY valid JSON matching this schema with these examples" is the difference between a 30% failure rate and a 2% failure rate in production.

Technique 1: Explicit Schema

The most reliable baseline is to provide an explicit schema showing the exact structure you expect. JSON Schema is the industry standard:

Extract customer details from the message below.
Respond with ONLY valid JSON matching this schema:

{
  "type": "object",
  "properties": {
    "name": { "type": "string", "description": "Full customer name" },
    "email": { "type": "string", "description": "Email address" },
    "phone": { "type": "string", "description": "Phone number (digits and hyphens only)" }
  },
  "required": ["name", "email"],
  "additionalProperties": false
}

Customer message: "Hi, I'm Sarah Chen. You can reach me at [email protected] or 555-0142."

This schema does several things:

• Defines exact field names and types.
• Marks phone as optional (not in required), so a missing phone number is valid.
• Forbids extra fields via "additionalProperties": false.
• Includes descriptions for edge cases (phone format).

The result is reliable, parseable JSON.

Technique 2: Concrete Examples

Schemas alone are good; schemas plus worked examples are better. Examples show the model how the boundary cases behave:

You are extracting product reviews into structured JSON.
Respond with ONLY valid JSON matching this format. If a field is missing, use null.

Example 1:
Input: "Great product! 5 stars. Bought on 2026-06-15."
Output: {
  "text": "Great product!",
  "rating": 5,
  "purchase_date": "2026-06-15"
}

Example 2:
Input: "Okay, but shipping was slow."
Output: {
  "text": "Okay, but shipping was slow.",
  "rating": null,
  "purchase_date": null
}

Now extract this review:
Input: "Not worth the price. 2 stars."
Output:

By showing concrete inputs and outputs, you teach the model the exact behavior you expect when data is sparse or missing. This reduces hallucination (the model won't invent a fake purchase_date) and ensures consistent parsing.

Technique 3: Native APIs — Function Calling and JSON Mode

Modern language model APIs offer features that go beyond prompting to enforce structure at the protocol level:

OpenAI's JSON Mode

import json
from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
  model="gpt-4o",
  temperature=0,  # Reduces hallucination
  response_format={"type": "json_object"},
  messages=[{
    "role": "user",
    "content": """Extract customer info as JSON with fields: name (string), email (string), phone (string or null).
    Customer: "Sarah Chen, [email protected], 555-0142"."""
  }]
)

data = json.loads(response.choices[0].message.content)
print(data)

With response_format={"type": "json_object"}, the API guarantees valid JSON output — the model cannot return prose. This removes one class of errors entirely.

Anthropic's Tool Use (Function Calling)

import anthropic
import json

client = anthropic.Anthropic()

response = client.messages.create(
  model="claude-3-5-sonnet-20241022",
  max_tokens=1024,
  tools=[{
    "name": "extract_customer_info",
    "description": "Extract customer details from text",
    "input_schema": {
      "type": "object",
      "properties": {
        "name": {"type": "string"},
        "email": {"type": "string"},
        "phone": {"type": ["string", "null"]}
      },
      "required": ["name", "email"]
    }
  }],
  messages=[{
    "role": "user",
    "content": "Extract: Sarah Chen, [email protected], 555-0142"
  }]
)

# Extract the tool use block
for block in response.content:
  if block.type == "tool_use":
    result = block.input
    print(json.dumps(result, indent=2))

Tool use (or function calling) is even more reliable than JSON mode because the model is constrained to fill in a defined tool's parameters. The output is guaranteed to match your schema, and you receive it as structured data, not a string to parse.

Google Gemini's Controlled Generation

import anthropic
import json

client = anthropic.Anthropic()

response = client.messages.create(
  model="claude-3-5-sonnet-20241022",
  max_tokens=1024,
  messages=[{
    "role": "user",
    "content": "Extract customer info (name, email, phone) from: Sarah Chen, [email protected], 555-0142. Return valid JSON."
  }]
)

(Google's API is similar but labeled differently in their documentation.)

Key takeaway: If your LLM provider offers native structured output or function calling, use it. It is more reliable than prompt engineering alone because it shifts the guarantee from "model will try to follow your format request" to "API will enforce the format."

Method Comparison: Prompt-Only vs. JSON Mode vs. Function Calling

| Dimension | Prompt + Schema Only | OpenAI JSON Mode | Anthropic Tool Use | Function Calling (Generic) | |-----------|---|---|---|---| | Reliability | 85–90% valid output | 99%+ valid JSON | 99%+ valid structure | 95–98% (varies by provider) | | Setup complexity | Low (write prompt) | Low (add parameter) | Low (define tool schema) | Medium (schema + integration) | | Cost | Baseline | Baseline | Baseline | Baseline + tool overhead | | Latency | Standard | Standard | Standard | Standard | | Error modes | Preamble, malformed JSON, type errors | None (guaranteed JSON) | None (guaranteed schema) | Schema drift, hallucinated fields | | Validation needed? | YES (always) | Minimal (JSON parsing only) | Minimal (schema validation only) | YES (full schema validation) | | Best for | Simple flat structures | Moderate complexity | Complex nested structures | Agent-based systems | | Testability | Moderate (manual tests needed) | High (deterministic output) | High (deterministic output) | Moderate |

Decision rule: Use function calling / tool use when available (99% reliability). Fall back to JSON mode if function calling not available (99% JSON validity, still need type checking). Use prompt + schema only when neither is available (require robust validation).

Real failure example: Why validation matters

# Function calling guarantees structure, but not meaning
response = client.messages.create(
  model="claude-3-5-sonnet-20241022",
  tools=[{
    "name": "extract_customer",
    "input_schema": {
      "type": "object",
      "properties": {
        "name": {"type": "string"},
        "email": {"type": "string", "format": "email"},
        "age": {"type": "number", "minimum": 0, "maximum": 120}
      },
      "required": ["name", "email"]
    }
  }],
  messages=[{"role": "user", "content": "Extract: John Doe, [email protected], age 25"}]
)

# This returns VALID JSON matching the schema, but what if input said "age -5"?
# Function calling validates type (number) but not the meaningful constraint (0-120).
# You still need application-level validation.

Technique 4: Multi-Step Extraction with Validation

For high-stakes structured output, combine prompting with validation:

Step 1: Extract the facts from the document.
Step 2: Format as JSON.
Step 3: Review your JSON for validity (all required fields present, types correct).

Document: ...

Your JSON output:

Then, on the client side:

import json
from jsonschema import validate, ValidationError

schema = {
  "type": "object",
  "properties": {
    "name": {"type": "string"},
    "email": {"type": "string"},
    "phone": {"type": ["string", "null"]}
  },
  "required": ["name", "email"]
}

try:
  data = json.loads(model_response)
  validate(instance=data, schema=schema)
  print("Valid:", data)
except (json.JSONDecodeError, ValidationError) as e:
  print("Invalid output. Retry or escalate:", e)
  # Optionally retry with corrected schema

This pattern ensures you catch malformed output before it propagates to your database or downstream systems.

Technique 5: Graduated Response Complexity

When extracting complex hierarchical data, scale up gradually:

Simple (flat):

{"name": "string", "email": "string"}

Moderate (nested):

{
  "customer": {"name": "string", "email": "string"},
  "order": {"id": "number", "total": "number"}
}

Complex (array of objects):

{
  "customer": {"name": "string"},
  "items": [
    {"id": "number", "quantity": "number", "price": "number"}
  ]
}

For each step up in complexity, provide more examples so the model understands the structure. A single flat schema might work with just a prompt, but nested or array structures often need 2–3 worked examples to get consistent output.

Complexity Example: Extracting a Multi-Level Invoice

You are extracting invoice data into JSON.
Every fact must exist in the source invoice; do not invent line items or charges.
Return ONLY valid JSON.

Example Invoice:
"Invoice #123 to Acme Corp, due 2026-07-30. Items: Widgets (qty 5, $10 each), Service Fee ($50). Subtotal $100, Tax $8, Total $108."

Example Output:
{
  "invoice_number": "123",
  "company": "Acme Corp",
  "due_date": "2026-07-30",
  "line_items": [
    {"name": "Widgets", "quantity": 5, "unit_price": 10, "total": 50},
    {"name": "Service Fee", "quantity": 1, "unit_price": 50, "total": 50}
  ],
  "subtotal": 100,
  "tax": 8,
  "total": 108
}

Now extract this invoice:
"Invoice ORD-2026-A456 billed to TechStartup Inc, due date November 15, 2026. Line items: Cloud hosting (1 month, $500), Premium support add-on (1 month, $100). Subtotal: $600. Tax (8%): $48. Total due: $648."

Expected output: Valid JSON matching the example structure, with values from the new invoice.

Why nesting is tricky: Without the nested structure example, a model might flatten it:

// WRONG (flattened)
{
  "invoice_number": "ORD-2026-A456",
  "company": "TechStartup Inc",
  "item_1_name": "Cloud hosting",
  "item_1_qty": 1,
  "item_2_name": "Premium support",
  "item_2_qty": 1
}

With the example showing "line_items": [...], the model understands the array structure and produces the correct nested output. Examples eliminate ambiguity that schemas alone cannot resolve.

Technique 6: Fallback Strategies When Native APIs Are Not Available

Not every provider or use case has native structured output. When you cannot rely on API-level enforcement, use a validation + retry loop:

import json
import anthropic

client = anthropic.Anthropic()

def extract_with_validation(prompt_text, schema, max_retries=3):
    for attempt in range(max_retries):
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt_text}]
        )

        try:
            data = json.loads(response.content[0].text)
            # Optional: Validate against schema
            # validate(instance=data, schema=schema)
            return data
        except json.JSONDecodeError:
            if attempt < max_retries - 1:
                # Retry with stricter instructions
                prompt_text += "\n\nReminder: Respond with ONLY valid JSON, no preamble."
            else:
                raise ValueError(f"Failed to extract valid JSON after {max_retries} attempts")

This pattern ensures that if the model deviates from the format once, you get a second (or third) chance to correct it.

Common Mistake

The mistake: Assuming that if you ask for JSON, you will reliably get valid JSON.

Extract the data as JSON.
Input: ...
JSON output:

Without an explicit schema or API-level enforcement, the model might return:

The extracted JSON is:
{
  "name": "John",
  "email": "[email protected]"
}

Notice the preamble "The extracted JSON is:" — this is not valid JSON. Your parser fails.

The fix: Enforce format with schema + API feature + validation.

Respond with ONLY valid JSON, no preamble or explanation.
{
  "type": "object",
  "properties": {
    "name": {"type": "string"},
    "email": {"type": "string"}
  },
  "required": ["name", "email"]
}

Input: John, [email protected]

And use the provider's JSON mode or function-calling API. Then validate the output before use:

try:
  data = json.loads(response)
except json.JSONDecodeError:
  raise ValueError("Model did not return valid JSON")

Real-World Pitfalls and Solutions

| Pitfall | Symptom | Solution | |---------|---------|----------| | Missing optional fields | Output omits some keys entirely instead of using null | Include example with null values; explicitly state "use null for missing data" | | Type confusion | IDs come back as strings instead of numbers, or vice versa | Schema must specify exact type; provide example with correct types | | Nested object flattening | Model returns {"customer_name": "X", "customer_email": "Y"} instead of {"customer": {"name": "X", "email": "Y"}} | Provide worked example of exact nesting structure | | Array truncation | Model stops outputting array items mid-way | Set explicit max items or chunk large arrays into multiple requests | | Preamble bloat | Output includes "Here is the JSON:" before the actual JSON | Prompt with "Respond with ONLY valid JSON, no explanation or preamble" | | Inconsistent string escaping | Quotes and newlines inside strings are not escaped correctly | Use triple quotes or HEREDOC in prompt to show exact escaping |

Production Validation Checklist

Before deploying structured-output extraction to production, test these scenarios:

1. Empty input: Does the model return null, "", or invent a value? Specify behavior explicitly.
2. Missing optional fields: Model should omit them or use null, not invent defaults.
3. Very long strings: Does the model truncate or preserve the full text?
4. Special characters: Test quotes, newlines, Unicode, HTML entities, escaped JSON strings.
5. Array handling: Does the model stop at array boundaries, or does it hallucinate extra items?
6. Type mismatches: If a field should be a number, what happens if the input contains words?
7. Boundary values: For numeric fields with min/max, test at and beyond boundaries.

Production example:

def validate_extracted_data(data, schema):
    """Validate extracted JSON against schema and business rules."""
    try:
        # 1. JSON parsing
        parsed = json.loads(data) if isinstance(data, str) else data
    except json.JSONDecodeError as e:
        return {"valid": False, "error": f"Invalid JSON: {e}"}

    # 2. Schema validation
    try:
        validate(instance=parsed, schema=schema)
    except ValidationError as e:
        return {"valid": False, "error": f"Schema violation: {e.message}"}

    # 3. Business rule validation
    if "email" in parsed and "@" not in parsed["email"]:
        return {"valid": False, "error": "Email missing @"}

    if "age" in parsed and not (0 <= parsed["age"] <= 120):
        return {"valid": False, "error": "Age out of valid range"}

    return {"valid": True, "data": parsed}

This three-layer validation (JSON → schema → business rules) catches 95%+ of real-world errors before they reach downstream systems.

Structured output is one of the highest-ROI prompting skills because it bridges language models (which naturally generate prose) and systems (which need precise formats). Use explicit schemas, provide concrete examples, leverage native API features (JSON mode, function calling), and always validate before trusting. Together, these techniques make structured extraction reliable enough for production use at scale — transforming what could be a 30% failure rate into a <2% failure rate. Your systems will thank you.