Continuous Delivery for Machine Learning

Three triggers, one pipeline

Continuous delivery for ordinary software rests on an assumption that quietly fails here: the only thing that changes is code. Commit, build, test, deploy. A green suite means the change is safe.

Machine learning systems have three independent change triggers.

Code changed. A new feature transform, a different architecture, a dependency bump.

Data changed. A fresh snapshot, a relabeled set, a new upstream source, a schema migration in a table you read.

Config changed. A decision threshold, a hyperparameter, a routing rule.

Any of the three can change the system's behavior without either of the others moving. A data refresh with zero code changes produces a genuinely different model. Your test suite passes, because your tests test code.

This is why the CI pattern needs adjusting rather than adopting wholesale. The pipeline must accept all three triggers, and the artifact each produces must clear the same evaluation before it can be promoted. Note what this rules out: a nightly retraining job that deploys on completion is not continuous delivery. It is continuous deployment of unevaluated artifacts, and it is how teams end up serving a model trained on a corrupted data snapshot.

The correct shape is that every trigger produces a candidate, and candidates pass through gates. The pipeline is fully automated; the promotion is a gate that a candidate either clears or does not.

The ML test pyramid

Most ML repositories have model quality tests and nothing underneath. That is an inverted pyramid, and it fails in the usual way: the expensive slow tests catch problems that cheap fast tests should have caught, hours later and with worse diagnostics.

Bottom layer — plain unit tests. Most of them. Feature transforms are pure functions. Test them like pure functions, with fixed inputs and expected outputs. Cover the boundaries that actually break: nulls, empty collections, timezone edges, negative values, unicode, the maximum length your string field permits. These run in milliseconds and catch the majority of skew bugs before anything is trained.

def test_days_since_signup_handles_same_day():
    # A same-day signup must be 0, not 1 and not negative.
    # This exact case caused a production skew incident: the serving
    # path rounded up and the training path truncated.
    features = transform_features(
        raw=RawInput(signup_ts=datetime(2026, 7, 30, 23, 50, tzinfo=UTC), order_value=None),
        as_of=datetime(2026, 7, 30, 23, 59, tzinfo=UTC)
    )
    assert features.days_since_signup == 0
    assert features.order_value == FITTED.order_value_mean

Second layer — data validation. Many. Every training run and every serving batch validates its inputs before use: schema conformance, value ranges, null rates, category cardinality, row counts, freshness. This layer catches the upstream changes that no code review will, because the change happened in someone else's system. A null rate moving from two percent to forty is a hard failure, not a warning.

Third layer — pipeline integration. Some. Train a deliberately tiny model end to end on a fixture dataset, then assert that the artifact serializes, loads in the serving container, and returns a well-formed prediction. This is not about quality; it is about the pipeline being wired together. It catches the serialization mismatch that would otherwise be discovered at deploy time.

Top layer — model behavior. A few. Slice metrics against thresholds. Invariance tests: a change that should not affect the prediction does not. Directional expectation tests: increasing a feature that should raise the score raises it. A small set of known failure cases from past incidents, frozen permanently.

The proportions matter as much as the presence. Behavior tests are slow, occasionally flaky, and give vague diagnostics — "f1 dropped" does not tell you where. Unit tests are fast and precise. Push as much coverage down the pyramid as you can.

The promotion gate

A candidate must clear four gates. Most teams implement the second and treat the others as paperwork.

Gate 1 — reproducible. Rebuild the candidate from its pinned inputs and confirm the metrics land inside a tolerance. This is not ceremony: an artifact you cannot rebuild is an artifact you cannot debug, patch, or explain. If your training is genuinely nondeterministic, define the tolerance explicitly and require the rebuild to land inside it.

Gate 2 — better on the evaluation set. Aggregate and per-slice, with the rule that no protected or business-critical slice may regress beyond its threshold. A candidate that gains two points overall while losing eight on new users is not an improvement; it is a redistribution, and someone should have to argue for it explicitly rather than have it hidden by an average.

Gate 3 — within operating limits. Latency at p99, memory footprint, cold start time, and cost per thousand predictions. A more accurate model that is three times slower may be unshippable, and it is far cheaper to learn that at the gate than after a capacity incident.

Gate 4 — owned and reversible. A named approver, a recorded rollback target, and a monitoring plan that says which metric will be watched for how long. This gate is not about the model. It is about whether anyone will notice when it degrades.

Encoding the gates in configuration keeps them honest, because a threshold in a file gets reviewed when it changes:

promotion_gates:
  reproducibility:
    rebuild_required: true
    metric_tolerance: 0.002

  quality:
    primary_metric: f1
    min_improvement: 0.005
    slices:
      # A slice regression beyond its threshold blocks promotion.
      # Overriding requires a recorded justification, not a rerun.
      new_users:   { max_regression: 0.01 }
      enterprise:  { max_regression: 0.01 }
      region_emea: { max_regression: 0.015 }

  operating_limits:
    p99_latency_ms: 200
    memory_mb: 4096
    cost_per_1k_predictions_usd: 0.12

  ownership:
    approver_required: true
    rollback_target_required: true

Automate the pipeline, not the decision

The ambition of a fully automated retraining loop is appealing, and the distinction that makes it safe is between automating work and automating judgment.

Automate every mechanical step: data validation, training, evaluation, artifact building, gate checking, shadow deployment, and metric collection. All of it should run without a human, on a schedule or on a trigger, and produce a candidate with a complete evidence package.

Then be deliberate about the promotion decision. For low-stakes models with strong gates and reliable rollback, automatic promotion on all gates passing is reasonable and is where mature teams end up. For models that touch money, safety, access, or anything legally consequential, a human approves — reviewing the evidence the pipeline produced rather than assembling it.

The intermediate position is the practical one for most teams: automatic promotion to shadow, automatic promotion to a small canary, human approval for full rollout. The automation handles everything up to the point where real exposure grows, which is the point where judgment has something to contribute.

A related trap is retraining on a schedule with no trigger logic. Retraining weekly whether or not anything changed burns compute, produces a stream of near-identical candidates that nobody reviews carefully, and creates gate fatigue. Retrain when something warrants it: enough new data, a measured drift threshold crossed, a data quality incident resolved, or a code change to the feature pipeline.

Worked example: a pipeline that caught a bad snapshot

A team retrains a routing classifier weekly. One Monday, the upstream ticket export runs while a schema migration is in progress. Roughly eight percent of records land with a null product_area — the classifier's strongest feature.

Without gates. Training runs. The model learns to route those tickets to a default queue. Aggregate accuracy drops from 0.91 to 0.89, which looks like ordinary week-to-week variance. The model deploys. Support tickets for one product area start routing to the wrong team. It is noticed nine days later by a team lead wondering why their queue is empty.

With gates. Data validation runs before training. The null rate for product_area is 8.2 percent against a configured maximum of 1 percent. The pipeline fails immediately with a precise message naming the field, the observed rate, and the threshold. No training compute is spent. An alert routes to the data platform team, who identify the migration, rerun the export, and the pipeline completes normally that afternoon.

The difference in cost is stark — nine days of misrouted tickets versus a two-hour delay — and the gate that made the difference is the cheapest one in the pyramid. A null-rate check is a few lines of configuration. It is also the layer teams skip, because it is not interesting and it does not feel like machine learning.

Worth noting what the quality gate alone would not have caught: a two-point aggregate drop sits comfortably inside normal variance for this model. Only the data validation layer, running before training, had the resolution to see the actual problem. That is the argument for the pyramid in one example.

Common mistake

The most common mistake is porting software CI unchanged and assuming a green suite means safety. Code tests validate code. They are silent about a data snapshot that arrived corrupted, a config threshold someone nudged, or a model that improved on average by getting worse for a segment you care about.

The second mistake is automating promotion before the gates are trustworthy. Automatic deployment on top of weak evaluation is a machine for shipping bad models quickly. Build the gates, watch them work for a few cycles, confirm they actually block things — and only then remove the human.

The correcting exercise: take your last five model deployments and ask, for each, which gate would have caught a problem if there had been one. If the honest answer for most of them is "an engineer looked at a number and it seemed fine," you have a review habit rather than a pipeline, and it will not survive the week that engineer is on vacation.