Package a Reproducible Model

What "reproducible" has to mean

Reproducibility gets discussed as though it were one property. It is really three tiers of inputs with very different failure rates, and teams reliably pin the first tier and skip the third.

The left column is what version control and an experiment tracker give you almost for free. Nobody loses their commit hash. The middle column is where the first real failures live: a data snapshot that was overwritten, a label definition that changed when someone fixed an upstream bug, a split that used the system random seed. The right column is where reproduction actually dies — a transitive dependency that floated a minor version, a different CUDA build, a machine with a different thread count.

This produces a practical consequence that is worth stating plainly: bit-identical retraining is not a realistic bar on GPU hardware, and pretending otherwise makes the check useless. Non-deterministic kernel scheduling and floating-point non-associativity mean two runs of the same command on the same machine can differ in the last decimal places, and those differences compound over epochs.

So do not assert equality. Declare a tolerance:

# reproduction-check.yaml
# Re-running training from a clean checkout must land inside these bands.
# Chosen from the observed spread across five same-seed runs, doubled.
tolerances:
  val_auc:        { expected: 0.8742, abs_tolerance: 0.004 }
  [email protected]: { expected: 0.6130, abs_tolerance: 0.010 }
  train_rows:     { expected: 1_284_402, abs_tolerance: 0 }   # data must match exactly
artifact:
  # The weights will differ; the interface and behavior on golden inputs must not.
  golden_predictions: tests/golden/predictions.jsonl
  max_prediction_delta: 0.01

Two things make this work. The tolerance is derived from measured variance rather than guessed, and the row count has a tolerance of zero — data drift is never acceptable noise, even when metric drift is.

The contract is wider than the model file

The most expensive reproducibility failures are not in training. They happen when a model that scored well offline behaves differently in production, and the cause is almost always in a row of this diagram that nobody thought of as part of the model.

Preprocessing is the usual culprit. A scaler fitted on training data, a tokenizer version, an imputation constant, a category encoding map. If these are reimplemented in the serving path rather than shipped with the artifact, they will drift apart. Not immediately — the first divergence is usually someone "cleaning up" a duplicated constant six months later.

Units are the silent one. A field that was seconds during training and milliseconds in the production event stream produces no error, no schema violation, and a model that is quietly wrong for everyone.

Failure behavior is the row that is almost always blank. What does the caller receive when the model times out, when the batch exceeds the ceiling, when the feature store is unavailable? "Wrong answer" and "no answer" require different handling downstream, and if you do not specify it, each caller will invent something different.

A contract worth having is executable:

class ScoreRequest(BaseModel):
    session_seconds: conint(ge=0, le=86_400)   # SECONDS. not ms.
    item_count:      conint(ge=0, le=500)
    country:         constr(regex=r"^[A-Z]{2}$")
    # Absent != zero. The imputation value is part of the artifact.
    referrer_domain: str | None = None

class ScoreResponse(BaseModel):
    score:         confloat(ge=0.0, le=1.0)
    model_version: str          # returned on every response, always
    degraded:      bool = False # True when a fallback path produced this

Returning model_version on every response costs nothing and is the single most useful field you will have during an incident, because it lets you answer "which version produced this bad output" without correlating timestamps against a deploy log.

Draw the release boundary

Experimentation and release need different rules, and most teams have no explicit moment where the rules change. That moment is the release boundary, and naming it prevents a specific failure: promoting something that was never evaluated.

Before the boundary, everything is disposable. Runs get deleted, environments drift, and a failed experiment costs an afternoon. This is correct — exploration should be cheap and unceremonious.

After the boundary, the artifact is immutable and checksummed, the evaluation report is versioned alongside it, and the environment is a locked image. A failure now costs an incident.

The crossing point is promotion, and promotion should name a checksum:

# Not this — "latest" is a moving reference and nobody knows what moved.
promote --model fraud-scorer --stage production --version latest

# This. The checksum already has an evaluation report attached to it.
promote --model fraud-scorer \
        --artifact sha256:9f2c1a…e07b \
        --eval-report reports/9f2c1a-eval.json \
        --approved-by g.okafor --ticket MLE-2291

The second form makes an important thing impossible: you cannot promote an artifact that has no report, because the command requires one. Constraints that are enforced by the tool survive; constraints that live in a runbook do not.

The model card is where claims get bounded

A model card is often treated as documentation overhead. Its actual function is narrower and more useful: it is where you write down what the model is not for, so that a future team does not discover the limits by shipping them.

The sections that earn their keep are intended use, prohibited use, evaluation sets with subgroup breakdowns, known failure modes, and the rollback target. Everything else can be generated. Link every claim to a versioned report rather than restating a number, because restated numbers go stale silently.

The subgroup breakdown deserves specific attention. An aggregate metric can hide a model that performs well overall and poorly for a segment that happens to be small in your test set and large in a market you are about to enter.

Practice: clean-room reproduction

The only way to know whether packaging works is to have someone else run it.

1. Train a small model on public data. Register parameters, metrics, environment, and artifact checksum.
2. Write the reproduction command and the tolerance file.
3. From a fresh checkout on a different machine — or a CI job with no cache — run the command.
4. Compare against the tolerances. Investigate anything outside them; widen a band only with a written reason.
5. Start the service locally. Send a valid request, a request with a missing optional field, a request with an out-of-range value, and a malformed body. Confirm each response matches the contract.
6. Hand the written command to someone who has not seen the project and watch them run it without asking you a question.

Step six is the real test, and it usually fails the first time on an unwritten assumption — an environment variable, a credential, a data mount.

Common mistake

The most common mistake is retraining during deployment. It appears in pipelines as a convenience: the deploy job pulls the latest data, retrains, and ships the result. It removes the release boundary entirely.

Now the artifact in production was never evaluated by a human, the evaluation report describes a different model, and rollback has no target because the previous artifact was never retained. A retraining pipeline should end by producing a candidate artifact and a report. Promotion is a separate, later decision — even when that decision is automated, it should be automated against gates rather than fused into the training job.