AI Supply Chain Security

Six layers, one owner missing

Software supply chain security is a mature practice. Most teams scan dependencies, pin versions, watch for advisories, and produce a bill of materials. Then they ship an AI feature and inherit five additional layers that none of that tooling covers.

Base model weights. A multi-gigabyte binary from a third party, loaded into your process, whose behavior you cannot inspect by reading it. It arrives from a hub where the publisher can update, gate, relicense, or delete it.

Training and tuning data. Corpora you did not assemble, with provenance you cannot verify, plus whatever internal data you fine-tuned on. The claim "trained on public web data" is a description of scale, not of curation.

Adapters and community checkpoints. LoRA weights and merged models, frequently uploaded by accounts with no organizational identity, downloaded because they scored well on a leaderboard.

Framework packages. The one layer your existing tooling does cover — the serving stack, tokenizers, inference runtime, and their transitive dependencies. Worth noting that this layer is unusually deep and moves unusually fast in the ML ecosystem.

Tool servers. MCP servers and plugins are third-party code that your agent invites into its own privilege context, often installed with a single command, frequently from a repository with a handful of stars.

Runtime content. Documents, pages, and API responses pulled in at request time. This is supply chain too: it is untrusted third-party content entering your system, just on a per-request basis rather than at build time.

The pattern worth noticing is that traditional dependency scanning covers exactly one of these six. The other five usually have no named owner, which is why they are where the interesting risk sits.

Treat model artifacts like packages

The build-time pattern that most teams start with looks harmless:

# Resolves at build time to whatever the hub is serving today.
model = AutoModel.from_pretrained("some-org/some-model")

This is the equivalent of pip install with no version, no lockfile, and no local mirror. Five questions expose what it costs.

Identity. Which exact revision are you running? A model name is a moving pointer. Pin the commit hash or digest, and record it in the artifact metadata of every build so that "which weights produced this output" is answerable later.

License. Many popular open-weight models carry custom licenses with field-of-use restrictions, acceptable-use clauses, or obligations that attach to outputs. "Open" in this ecosystem does not reliably mean what it means for code. Read the actual license, and check whether it constrains your sector or your commercial use.

Origin. Who published it, and can the upload be tied to that publisher? Typosquatting works on model hubs the same way it works on package registries, and a model named to resemble a well-known one, with a plausible model card, is cheap to publish.

Integrity. Is the artifact checksummed and mirrored into infrastructure you control? Pulling weights from a public hub during a production build means your deployment depends on a third party's uptime, their access controls, and their decision not to change the file.

Continuity. What happens if it is deleted, gated behind an agreement, or silently replaced tomorrow? For hosted API models, the analogous question is deprecation: what is your plan when the version you validated is retired on a published timeline?

The remediation is unglamorous and effective — the same thing you already do for packages:

# Pinned revision, verified digest, served from an internal mirror.
# A model change is now a reviewable commit, not a silent build difference.
MODEL_REVISION = "e3b0c44298fc1c149afbf4c8996fb924"

model = AutoModel.from_pretrained(
    "internal-mirror/some-org/some-model",
    revision=MODEL_REVISION,
    local_files_only=True  # builds fail loudly rather than fetching something new
)

local_files_only=True is the part that matters most. It converts a silent substitution into a build failure, which is exactly the trade you want.

Why poisoning evades your metrics

Data and model poisoning gets discussed as an exotic threat. The mechanism is worth understanding precisely, because it explains why standard quality checks cannot detect it.

An attacker introduces crafted examples into a corpus — a fine-tuning set, a scraped dataset, a RAG knowledge base, or a public source that feeds one of these. The examples associate a rare trigger with a chosen behavior. Everything else about the model is left intact, deliberately.

Now consider what your evaluation measures. Aggregate accuracy on a held-out set is unchanged, because the poisoned behavior fires only on the trigger, and the trigger does not appear in your evaluation data. Loss curves look normal. Human spot-checks look normal. The model is, by every metric you compute, fine.

The gap between injection and observed effect is measured in months. By the time the behavior is triggered in production, the corpus has been rebuilt several times, the artifact has been retrained, and reconstructing which examples were present at which point is either a lineage query or an archaeology project.

Three controls change the situation, and none of them is detection-based:

Source authorization. Maintain an explicit list of what may enter a training or retrieval corpus. Data arriving from an unlisted source is rejected rather than sampled. For RAG this is the highest-value control by a wide margin — most corpus poisoning is not sophisticated, it is a document that nobody authorized being ingested by a crawler nobody scoped.

Lineage. Every training example and every indexed chunk should be traceable to a source, a timestamp, and an ingestion job. When something is discovered, the question "what else came from that source" must be answerable by a query.

Targeted evaluation. Aggregate metrics cannot find this, so evaluate on the specific behaviors that would matter if subverted. If the model must never recommend a competitor, test that explicitly and repeatedly rather than trusting that a general quality score covers it.

For RAG specifically, there is a fourth control that outperforms the rest: require citations and verify that the cited chunk actually supports the claim. A poisoned chunk that must be shown to the user alongside the answer is far more likely to be noticed than one that silently shapes a paragraph.

Tool servers are dependencies with privileges

MCP servers and agent plugins deserve separate attention, because their install experience is deceptively casual and their privilege is unusually high.

Adding one is typically a single line in a configuration file. What that line does is start a process that your agent will call, which can read the arguments the agent sends, return content that enters the agent's context window, and — depending on the server — reach your filesystem, your network, and your credentials.

Two attack shapes matter here.

The server is malicious or compromised. It behaves normally for weeks, then begins returning content designed to steer the agent, or exfiltrates the arguments it receives. Because tool results enter the context window as trusted-looking content, a compromised server is a prompt injection source with a permanent seat at the table.

The server is honest but over-permissioned. More common by far. A filesystem server intended for one project directory is configured with a broader root. A database server is given the credentials that were already lying around. No attacker is required; a confused agent is sufficient.

Apply the same gates you would to any dependency, plus one:

• Pin the version, and read the changelog before upgrading. Auto-updating tool servers is auto-updating the code your agent runs.
• Run it with the narrowest filesystem and network scope that works, and verify the scope empirically rather than trusting the configuration comment.
• Give it a dedicated, minimally-scoped credential, never a shared one.
• Review the tool descriptions it advertises. Those descriptions go into your model's context, and a server can use them to influence agent behavior — a "tool poisoning" path that has no analogue in ordinary dependencies.

That last point is easy to miss and worth restating: a tool server controls text that enters your model's context window every single turn. That is a more intimate position than most dependencies occupy.

Worked example: an inventory that found three things

A team runs a support assistant. It uses a hosted API model, a fine-tuned classifier for routing, a RAG index over documentation, and three MCP servers. They spend an afternoon on an inventory.

Layer 1, base model. Hosted API, pinned to a dated version string. Contract confirms no training on their data. Deprecation risk is real, so they record the version in every response log and add a quarterly review of the provider's deprecation notices. No finding.

Layer 2, fine-tuning data. The routing classifier was fine-tuned eighteen months ago on exported support tickets. Nobody can say which export, and the engineer who ran it has left. They cannot prove which customer data is embedded in those weights, which means they also cannot honor a deletion request against it. Finding: retrain from a snapshot with recorded lineage.

Layer 3, adapters. None in use. No finding.

Layer 4, packages. Covered by existing scanning, though the inference runtime is three minor versions behind and two advisories apply. Ordinary patching work.

Layer 5, tool servers. Three servers. One is an internal build. One is a well-maintained project pinned to a version. The third was installed during a hackathon from a repository with eleven stars and has not been updated in a year — and it holds a database credential with write access, because that credential was already in the environment. Finding: remove it, or fork, review, and re-scope the credential to read-only.

Layer 6, runtime content. The RAG index ingests from a documentation site and — discovered during this exercise — from a shared drive folder that anyone in the company can write to. Any employee can silently change what the assistant tells customers, with no review step. Finding: restrict the ingestion source, or add review before indexing.

Three findings, one afternoon, and two of them were invisible to every scanner the team runs. That ratio is typical. The value of the inventory is not that it is sophisticated — it is that it forces someone to look at the five layers that have no owner.

Common mistake

The most common mistake is assuming existing dependency scanning covers this. It covers layer four. The other five layers are outside its model entirely, and their absence from the scan report reads as safety.

The second mistake is treating a model hub as a trusted registry. It is a hosting platform where anyone can publish. The same skepticism you apply to an unfamiliar npm package with few downloads applies to an unfamiliar checkpoint with a nice model card — plus the additional problem that you cannot read the weights to see what they do.

The correcting exercise is the inventory above. List the six layers, name an owner for each, and answer the five provenance questions for every model artifact you ship. It takes an afternoon and it consistently finds at least one thing that nobody knew was there.