Threat-Model an LLM Application

Draw the real system

Include users, application server, model provider, system instructions, retrieval stores, uploaded files, external pages, tools, credentials, logs, administrators, and downstream systems. Mark where data changes trust level and which component enforces each policy.

Treat user input, retrieved content, tool output, model output, and third-party metadata as untrusted. A model cannot reliably separate instructions from data by intention alone.

Write abuse cases

Test direct and indirect prompt injection, sensitive-data requests, cross-user data exposure, insecure output rendering, excessive tool permissions, confused-deputy behavior, poisoned retrieval, denial-of-wallet, dependency compromise, and misleading citations.

For an agent, ask what happens if the model selects the wrong tool, repeats an action, invents arguments, follows instructions embedded in a document, or continues after partial failure.

Layer controls

Use least-privilege credentials, schema validation, allowlists, deterministic authorization outside the model, data minimization, output encoding, sandboxing, rate and cost limits, confirmation for high-impact actions, audit logs, and revocation. Model instructions are useful but are not an authorization boundary.

Practice: adversarial test matrix

Choose three relevant OWASP risk areas. For each, define precondition, attack input, expected safe behavior, observable evidence, control owner, and regression test. Run the tests against both normal and adversarial documents. Record residual risk.

Incident readiness

Prepare kill switches, credential rotation, affected-user notification, evidence retention, and rollback. A mitigation that cannot be monitored or operated during an incident is incomplete.