Sensitive Information Disclosure in LLM Systems

Leakage is a plumbing problem

When teams audit an AI feature for data exposure, they almost always audit one thing: what goes into the prompt. That is the visible path, and it is genuinely important. It is also roughly one sixth of the surface.

Sensitive data leaves an LLM system through six routes, and they have different owners, different lifetimes, and different blast radii. A program that closes only the first will pass its own review and still leak.

Into the prompt. The direct path. A developer needs "customer context," so the whole record is serialized into the system message — including fields no part of the task uses.

Into the logs. Nearly every LLM integration logs full request and response bodies during development, because debugging a model without seeing its input is miserable. That logging is rarely removed, frequently ships to a third-party observability vendor, and inherits a default retention of months to years. This is the single most common real-world exposure, and it is created by engineers acting reasonably.

Into the vector store. RAG indexes chunk text alongside embeddings. If your source documents have per-document access rules and your index does not carry them, you have created a copy of your sensitive corpus with the permissions removed. Embeddings themselves are also not anonymous — treat an embedding of a document as a derivative of that document, not as a safe hash of it.

Into the cache. Semantic or prompt caches keyed only on request similarity will happily serve one tenant a response generated for another. This produces a spectacular failure mode: the leak is invisible in your code path because no query ran at all.

Into the error. Exceptions and provider error responses echo the offending payload. A 400 from a model provider containing the full prompt often lands in an error tracker with a different retention policy and a different access list than your application logs.

Into the next model. Whether conversations may be used for training is a contractual question, not a technical one, and the default differs by provider and by plan tier. It needs to be verified in writing and re-verified when your plan changes.

Enumerate all six for your own system before you claim data minimization. The exercise usually takes an hour and usually finds something.

The minimization ladder

Once you know where data can go, the next question is how much needs to go there at all. Most teams answer this backwards: they start with the full record and remove fields when someone objects. Invert it.

Rung 1 — do not send it. The strongest control, and the most under-used. For each field, ask what would change in the output if it were absent. In a "summarize this customer's recent issues" task, the customer's date of birth, full address, and payment method change nothing. They were included because the record was fetched whole.

Rung 2 — send a derived value. The model often needs a property of the data, not the data. An age band rather than a birth date. A region rather than a street address. A tier rather than a contract value. Derived values usually preserve the model's reasoning ability completely.

Rung 3 — send a token. Replace an identifier with a placeholder your application can resolve after the call. The model reasons about CUSTOMER_A and CUSTOMER_B; your code maps them back when rendering. This preserves relational reasoning across a conversation while keeping real identifiers out of the provider, the logs, and the cache.

Rung 4 — send it redacted. Mask the value but keep its shape, so the model still understands the field exists and what type it is. Useful when the model must reason about presence or format.

Rung 5 — send it raw. Sometimes genuinely necessary. When it is, it should carry a named owner, a documented justification, a retention limit, and a logged decision — not simply be the default because it was easiest.

A practical implementation is to make the boundary explicit in code rather than leaving it to whoever writes the next call site:

# One place decides what may cross the boundary to a model provider.
# Adding a field here is a reviewable event, not an incidental import.
MODEL_SAFE_FIELDS = {"account_tier", "region", "open_ticket_count", "signup_year"}

def to_model_context(customer: dict) -> dict:
    # Fail closed: unknown fields are dropped rather than passed through,
    # so a new column in the source table cannot silently start leaking.
    return {key: value for key, value in customer.items() if key in MODEL_SAFE_FIELDS}

The value here is not the filtering logic, which is trivial. It is that the allowlist is a single reviewable artifact. When someone needs to add email to it, that becomes a conversation instead of a one-line change buried in a feature branch.

Tenant isolation belongs in the query

The most damaging leaks in multi-tenant AI products come from retrieval that searches too broadly and filters too late.

The broken pattern is seductive because it looks correct in a diagram: search the index, get the top matching chunks, then check permissions before showing the answer. The problem is that "before showing the answer" is several steps too late. By then the other tenant's text has been placed in a context window, sent to a provider, written to your logs, possibly stored in a cache, and used to shape a response whose wording may reveal the content even after the citation is stripped.

There is also a subtler failure: filtering after retrieval silently degrades quality. If eight of your top ten chunks belong to other tenants and get discarded, the user receives an answer built from two weak chunks, and nothing in your metrics explains why the feature feels unreliable.

The correct pattern scopes the search itself. The tenant identifier is part of the query, enforced by the retrieval layer:

def retrieve(query: str, principal: Principal, k: int = 8):
    # The filter is a query parameter, not a post-processing step.
    # A caller cannot forget it: principal is required to build the request.
    return index.search(
        vector=embed(query),
        filter={"tenant_id": principal.tenant_id, "acl": {"$in": principal.group_ids}},
        top_k=k
    )

Two properties make this hold. The function cannot be called without a principal, so there is no path that searches unscoped. And the filter runs inside the index, so unauthorized chunks are never materialized anywhere in your process.

For document-level access rules that change over time, store the ACL on the chunk record and re-check at query time rather than baking permissions into a static partition at index time. Permissions change more often than documents do, and a stale partition is a leak with a delay fuse.

Retention: the part that outlives the feature

Minimization decides what enters the system. Retention decides how long it stays, and it is where most programs are weakest, because retention defaults are set by tools rather than by policy.

Write down, per store, the answer to three questions: what is kept, for how long, and who can read it.

The deletion path deserves particular attention because it is the one people discover they lack only when a customer exercises a deletion right. If a customer's document is deleted from the source system, what removes its chunks from the vector index, its text from the cache, and its content from ninety days of request logs? If the answer is "nothing automatic," that is a finding worth raising now rather than during a regulatory request.

Worked example: an internal HR assistant

A concrete case, because these controls interact in ways that are easier to see end to end.

An internal assistant answers employee questions from an HR document corpus. The corpus contains a public handbook, team-level policy documents, and individual compensation records. Three sensitivity levels, one index.

Minimization. The assistant needs the asking employee's team and tenure to answer policy questions correctly. It does not need their salary, their manager's name, or their home address. The to_model_context allowlist contains four fields. Compensation figures are never placed in a prompt; when a question requires one, the application answers from a template rendered outside the model.

Retrieval scoping. Each chunk carries a sensitivity label and an ACL. The query filter is built from the authenticated employee's identity, so a query for "what is the parental leave policy" searches the handbook and their team's documents, and cannot reach another team's documents or any compensation record.

Logging. Request logs record a request id, the retrieved chunk ids, token counts, latency, and the model version. They do not record chunk text or the completion. A separate, restricted, seven-day store captures full traces for one percent of requests for debugging, and access to it is audited.

Cache. The semantic cache key includes the employee's group memberships. Two employees on different teams asking identical questions do not share a cache entry, because they are not entitled to the same source material.

Deletion. When an employee leaves and their records are purged, a job removes their chunks from the index and their entries from the debug trace store. The job is tested quarterly with a synthetic record, because an untested deletion path is an assumption.

Now the interesting part: notice how little of this is AI-specific. It is access control, allowlisting, log hygiene, cache keying, and retention — the same disciplines you would apply to any system handling this data. That is the point. The novel part of AI security is the prompt injection channel; the leakage part is mostly classical engineering applied to a new set of data stores that nobody has inventoried yet.

Common mistake

The most common mistake is auditing the prompt and declaring victory. A team spends two weeks minimizing what goes into the model, ships, and never looks at the observability stack that has been recording every full request and response to a third-party vendor since the first prototype.

The second mistake is trusting the model to keep a secret you put in its context. Instructions like "do not reveal the salary figures below" are a preference, not a control. If a value must not reach a user, it must not reach the model — a rule that survives every jailbreak, every injection, and every future model upgrade.

The correcting habit is to inventory the stores rather than the code. Ask where a single sensitive field could physically exist thirty days after one request, then go look in each of those places and see whether it is actually there. The gap between the expected answer and the observed one is your finding list.