Experiment Tracking and Model Lineage

Five inputs, or it is not reproducible

"We track our experiments" usually means a table of parameters and metrics. That table tells you which run scored best. It does not tell you how to build that run again, which is the actual requirement.

A run is reproducible only when five inputs are pinned. Miss any one and you have a historical record rather than a recipe.

Code revision. The exact commit — plus a flag for whether the working tree was dirty. A run launched from uncommitted changes is unreproducible by construction, and the honest move is to record that fact rather than log the commit it was nearly at.

Data version. A snapshot identifier or content hash, never a live table reference. "Trained on the orders table" is not a data version, because the orders table today is not the orders table from March. This is the input teams most often skip and most often need.

Environment. A lockfile with resolved transitive versions, and ideally a container image digest. A minor version bump in a numerical library can move a metric enough to change a decision.

Parameters. Every value that affects the result, including the random seed. Seeds are frequently omitted because they feel like noise, and then a rerun differs and nobody can tell whether that is variance or a real change.

Hardware. Device type, device count, and numeric precision. Mixed-precision training on a different accelerator generation produces different numbers. Not usually decision-changing, but you want it written down when it is.

The output side has one rule: the artifact gets a checksum, and the deployment references that checksum. A deployment pointing at models/latest has erased the link between what is running and what was tested.

Record runs, not scores

Compare two entries in a tracking system.

run_37   f1=0.847   lr=3e-4

This is a scoreboard entry. It ranks runs. When someone asks in four months why the model behaves oddly on new accounts, it offers nothing: you cannot rebuild it, you cannot see which slice it was weak on, and you cannot tell what differed from run_36.

commit      = a3f91c        dirty       = false
data        = snap-2026-07-02
env         = sha256:7bd2...  device      = 1x A100, bf16
seed        = 17            lr          = 3e-4
f1          = 0.847         f1_new_users = 0.611
artifact    = sha256:c81e...

This is a run record. It is rebuildable, comparable, and honest about the slice where it is weak — and that last property is what makes it useful rather than merely complete. f1_new_users = 0.611 against an aggregate of 0.847 is the kind of fact that decides whether a model should ship, and it is invisible on a scoreboard.

Log slice metrics from the first run, not after the first incident. The slices worth tracking are usually obvious in advance: new versus established users, each major region, the largest customer segments, and any group where a regression would be a compliance problem. Adding them later means you cannot compare against history, which is precisely when you want to.

A practical convention that costs little and pays repeatedly:

# Log the identity of every input alongside the metrics.
# The dirty flag is deliberate: a run from uncommitted code is
# recorded as unreproducible rather than quietly mislabeled.
mlflow.log_params({
    "git_commit": git_commit(),
    "git_dirty": working_tree_dirty(),
    "data_snapshot": snapshot_id,
    "env_digest": image_digest(),
    "seed": seed,
    **hyperparameters
})

mlflow.log_metrics({"f1": f1, **{f"f1_{name}": value for name, value in slice_f1.items()}})
mlflow.log_artifact(model_path)  # checksummed by the tracking server

Design lineage around incident questions

The test of a lineage system is not how much it stores. It is how quickly it answers the four questions that arrive under pressure.

"Which model produced this bad prediction?" Requires the model version to be stamped on every prediction at serve time and carried into your logs. Without it, a customer complaint about an output from three weeks ago cannot be traced to an artifact, and you are debugging a model you may no longer be running.

"Was this customer's deleted record in the training set?" Requires data snapshots with content addressing. If training reads a live table, you cannot reconstruct what it saw, which makes deletion requests unanswerable for any model trained before the deletion.

"What was running before the regression?" Requires a deployment history keyed to artifact digests, with timestamps. Surprisingly often this exists only as a Slack thread.

"Who approved this and on what evidence?" Requires the promotion decision to link to a specific evaluation report, which links to a specific artifact. An approval that references "the new model" has recorded a feeling.

Design backwards from these four questions and the storage decisions become obvious. Design forwards from "log everything" and you will accumulate volume without answers.

Connect the artifact to the decision

The last link is the one that breaks most often: between the artifact running in production and the evidence that justified it.

The chain should be traversable in both directions. From a running service you should reach the artifact digest, then the run that produced it, then the five pinned inputs, then the evaluation report, then the approval. From a training run you should be able to see whether it ever reached production and when.

Two anti-patterns break this chain, and both look like convenience:

Rebuilding at deploy time. A pipeline that retrains as part of deployment produces an artifact nobody evaluated. Whatever was approved, it was not this. Promote artifacts, never recipes.

Floating tags. model:latest or model:prod are pointers, and pointers move. If the tag moved and something broke, you need to know what it pointed at before. Tags are fine as a convenience layer as long as the deployment record captures the digest the tag resolved to at the moment of deployment.

Worked example: tracing a Tuesday morning complaint

A customer reports that a pricing model quoted an absurd figure on 14 July. Walk the chain in a system that has these pieces, and then in one that does not.

With lineage. The request log for that transaction carries model_version=sha256:c81e. The registry maps that digest to run_412. The run record shows commit=a3f91c, data=snap-2026-06-28, seed=17, and slice metrics. Two things stand out immediately: the run was trained on a data snapshot taken before a pricing schema migration, and f1_enterprise was already weak at 0.58. The deployment history shows this artifact went live on 2 July, replacing sha256:9ff1. The approval links to an evaluation report where someone waived the enterprise slice regression because the aggregate improved.

Elapsed time: about ten minutes, most of it reading. The finding is not a mystery bug, it is a documented decision that turned out to be wrong — which is a far better outcome, because it points at the promotion gate rather than at the code.

Without lineage. The request log has no model version. The team checks what is currently deployed, but it may have changed since 14 July. The training code has moved on eight commits. Training reads a live table, so the data as of late June cannot be reconstructed. Nobody can say whether the enterprise slice was ever measured. The investigation becomes an argument about what probably happened.

The difference in outcome is not the sophistication of the tooling. Both teams may use the same tracking server. The difference is that one of them logged the five inputs and stamped the version on every prediction.

Common mistake

The most common mistake is tracking metrics without tracking inputs. The tracking server fills with runs, the leaderboard is satisfying, and not one of those runs can be rebuilt. Metrics are the output of an experiment; the inputs are the experiment.

The second mistake is treating tracking as a research tool that stops at the model handoff. The lineage chain has to extend through the registry, the deployment, and every served prediction. A perfect experimental record that ends at "we picked run 412" leaves the production half of every incident question unanswerable.

The correcting exercise: pick a model currently in production and try to answer the four incident questions using only recorded data, with a timer running. Anything that takes more than a few minutes, or requires asking a specific person, is a gap — and the person is a single point of failure, not a lineage system.