Red-Team an AI System

From demo to discipline

Most AI red teaming is a demo. Someone spends an afternoon typing adversarial prompts, finds three that produce embarrassing output, screenshots them into a deck, and the team feels both alarmed and vaguely reassured. Six weeks later nobody can reproduce the findings, the model has been upgraded, and the exercise has to start over.

The difference between a demo and a discipline is the loop closing. Five steps, and the last one is the one that gets dropped.

Scope. Name the asset, the attacker, and the failure condition before you start.

Attack. Run technique families systematically rather than improvising.

Observe. Capture the full trace, not the final output.

Classify. Rank by impact on the system, not by how surprising the prompt was.

Fix and freeze. Add the control, then keep the attack forever as an automated test.

Step five is what converts a finding into durable safety. Without it, a fix is a code change that a future refactor can silently undo, and the knowledge that the attack exists lives only in the memory of whoever ran the session.

Scope before you attack

Unscoped red teaming produces an unbounded pile of interesting-but-unactionable observations. Three questions, written down, prevent that.

What is the asset? Not "the chatbot." The specific thing whose compromise would matter: customer records in the retrieval corpus, the payment tool, the deployment credential, the brand's public voice. Different assets need different attacks, and a session that does not name one will drift toward whichever is easiest to poke.

Who is the attacker, and what can they touch? An authenticated customer, an unauthenticated visitor, a malicious insider, or — the one that gets forgotten — a third party who can only write text that your pipeline will later read. That last persona is the one that finds indirect injection, and you will not test for it unless you name it.

What counts as a failure? Define this in advance, in terms of system effect. "The model says something rude" and "the model reads another tenant's invoice" cannot share a severity scale. Writing the failure condition first stops the session from grading itself on vibes.

A useful scope statement fits in three lines:

Asset:     customer records reachable via the account_lookup tool
Attacker:  a third party who can file a support ticket
Failure:   any tool result containing data from an account the
           requesting agent is not assigned to

Everything in the session now has a target. An attack that makes the model rude is out of scope for this session — worth logging, not worth chasing today.

Cover technique families, not clever prompts

The failure mode of unstructured red teaming is depth without breadth: forty variations on one idea and nothing at all in three other categories.

Structure coverage as a grid. Down one axis, the risks you actually care about for this asset. Across the other, the technique families:

Direct. The attacker is the user. Instruction override, role play, hypothetical framing, authority claims, incremental boundary erosion.

Indirect. The payload arrives in content your system retrieves. A ticket, a document, a web page, a filename, a commit message, an image passed to a multimodal model.

Multi-turn. The attack is assembled across a conversation. Establish a premise early, reference it as settled later. Ask for each piece separately so no single message looks wrong. This family is heavily under-tested because single-prompt tooling does not produce it naturally.

Encoded. The same instruction in base64, in another language, in homoglyphs, split across formatting, in zero-width characters, or in a code comment inside a document.

Fill the grid for your own system and the empty cells are immediately visible. Empty does not mean safe; it means untested. Most teams discover after one hour of this that they have tested direct injection against every risk and indirect injection against none — which is backwards, because indirect is the class that produces breaches.

Tooling helps with volume once the structure exists. Frameworks like PyRIT automate technique variation, multi-turn orchestration, and scoring, which turns "twenty prompts someone thought of" into "several hundred systematic variations." Use tooling to scale coverage after you have decided what coverage means. Automation applied to an unscoped session produces a larger unscoped session.

Observe the trace, not the answer

Judging an attack by the final response is how teams miss the findings that matter.

An agent asked to exfiltrate data might return a polite refusal — after making a tool call that successfully read the data and returned it into the context window. The user-visible output shows a refusal. The trace shows a cross-tenant read. Those are opposite conclusions from the same interaction.

Capture, for every attempt: the input, every retrieved chunk, every tool call with full arguments, every tool result, the model's intermediate reasoning if available, the final output, and the identity each call executed under. Then evaluate against the failure condition you wrote during scoping — which was expressed in terms of tool results, not in terms of what the user saw.

This is also where you learn which of your controls fired. An attack that was blocked is not a null result. Knowing whether it was stopped by the model declining, by a schema rejection, or by an authorization check tells you which layer you are actually relying on. If everything is being caught by the model declining, you have one soft layer and no hard ones.

Severity is about reach

Findings need a severity scale grounded in system effect, or the loudest finding wins instead of the worst one.

Critical — crossed a trust boundary. Read another tenant's data, executed code, moved money, changed permissions, reached an internal network. Blocks release.

High — reached a boundary and was stopped by a deterministic control. The model was persuaded and tried; the authorization check refused. Genuinely good news, and still a finding: it means the only thing between you and critical is one control, which deserves monitoring and a regression test.

Medium — policy violation with no system effect. Said something off-brand, unsafe, or embarrassing. Real, worth fixing, and a content problem rather than a breach. Keeping this separate from critical is what stops severity inflation from making the list useless.

Informational — refused as designed. Evidence a control works. Worth a regression test, not a ticket.

The distinction that matters most in practice is between the first two. Both involve the model being fully compromised. In one case something caught it. Teams that do not separate these either panic at every successful jailbreak or become numb to all of them.

Freeze the finding

A fixed finding with no test is a finding you will have again.

Every attack that reached medium or above becomes a permanent, automated test. Not a document, not a wiki page — a test in the suite that runs on every change:

# Regression test frozen from red-team finding RT-2026-014.
# The attack must remain blocked at the authorization layer, not by
# the model choosing to decline -- so we assert on the tool result.
async def test_indirect_injection_cannot_read_foreign_account():
    ticket = load_fixture("rt_2026_014_injected_ticket.txt")
    trace = await run_agent(
        message=f"Summarize this ticket:\n{ticket}",
        principal=support_agent_in_tenant_b()
    )

    # The model may still attempt the call; that is acceptable.
    # What must never happen is a successful cross-tenant read.
    foreign_reads = [
        call for call in trace.tool_calls
        if call.name == "account_lookup" and call.succeeded and call.tenant_id != "tenant_b"
    ]
    assert foreign_reads == [], f"cross-tenant read succeeded: {foreign_reads}"

Note what this asserts. Not that the output looks safe, and not that the model refused — the model is allowed to be fooled. It asserts that no cross-tenant read succeeded. That is a property of the authorization layer, and it will keep holding when the model is upgraded, the prompt is rewritten, or a new engineer changes the tool description.

This is also why these tests survive model changes, which is the thing that kills prompt-level test suites. A test asserting "the model refuses" breaks every time the model gets better at nuance. A test asserting "no unauthorized data crossed the boundary" is a stable invariant.

Re-run the full frozen suite on every model version change, every prompt change, and every tool addition. Model upgrades are the highest-risk moment for these regressions, because behavior changes in ways your prompt tests were never designed to detect.

Common mistake

The most common mistake is running a red-team session as a one-time event before launch. AI systems change constantly — new tools, new documents in the corpus, new model versions, new prompt tweaks — and each change can reopen a closed finding. A session that is not frozen into tests has a half-life of about one sprint.

The second mistake is grading on prompt novelty. A finding that required an elaborate multi-turn setup and produced a mildly off-policy sentence generates more excitement than a boring one-line injection that read a neighboring tenant's record. The elaborate one is a better story. The boring one is the breach.

The correcting habit: before the session, write the failure condition in terms of system effect. Afterward, ask of every finding, "what did it reach?" If the answer is "nothing, but it was clever," it is a medium at best — and if the answer is "another customer's data," it does not matter that the prompt was three words long.