Training-Serving Skew

Four origins, three of them are bugs

Training-serving skew is the gap between the inputs a model learned from and the inputs it receives in production. It is the most common reason a model that validated beautifully performs poorly after launch, and it is badly served by being discussed as one phenomenon, because it has four origins that need four different responses.

Code skew. Training computes a feature one way; serving computes it another. Two implementations, written by different people at different times in different languages, that were meant to agree.

Time skew. A training feature was computed using information that did not exist at the moment being predicted. The model learned from the future.

Source skew. Training reads the warehouse; serving reads a live API. Same conceptual field, different null semantics, different rounding, different timezone handling, different staleness.

Distribution skew. The world changed. Live inputs no longer resemble the training distribution. This is the only one of the four that is not a defect — it is a fact about the world that you monitor and respond to.

Notice the ratio. Three of the four are engineering defects present on day one, shipping silently behind excellent offline metrics. Yet almost all monitoring effort goes to the fourth. Teams build drift dashboards while their serving path quietly computes a different feature than the one the model was trained on.

The diagnostic that separates them: if the model performed poorly from the first day in production, it is code, time, or source skew. If it performed well and degraded over weeks or months, it is distribution skew. That single question routes the investigation correctly and saves days.

One definition, two callers

Code skew has one durable fix, and it is not "be careful."

The classic shape:

# Training, in the feature pipeline
df["days_since_signup"] = (df["event_ts"] - df["signup_ts"]).dt.days
df["order_value"] = df["order_value"].fillna(df["order_value"].mean())

// Serving, in the request path, six months later
long daysSinceSignup = ChronoUnit.DAYS.between(signupTs, Instant.now());
double orderValue = order.getValue() != null ? order.getValue() : 0.0;

Three divergences hide in those four lines. Imputation differs — training uses the column mean, serving uses zero. Time reference differs — training uses the event timestamp, serving uses now. Timezone and rounding behavior between the two date libraries differ at day boundaries.

Each looks reasonable in isolation. Together they mean the model receives a feature it never saw during training, and no test fails, because both code paths do exactly what their authors intended.

The fix is structural: one definition, two callers.

# Versioned with the model artifact. Both the training job and the
# serving path import this exact function -- not a reimplementation.
def transform_features(raw: RawInput, as_of: datetime) -> Features:
    return Features(
        # `as_of` is required, never defaulted to now(). Training passes the
        # event timestamp; serving passes request time. Same code, same result.
        days_since_signup=(as_of - raw.signup_ts).days,
        # Imputation constants are baked into the artifact at fit time,
        # so serving cannot invent its own fallback.
        order_value=raw.order_value if raw.order_value is not None else FITTED.order_value_mean
    )

Two design choices carry the weight. as_of is a required parameter, so serving cannot silently substitute the current time. And imputation constants are fitted values stored with the artifact, so a change to imputation strategy is a model change requiring a new artifact — which is exactly what it is.

Where the serving path genuinely cannot run the training language, a feature store is the standard answer: define the transformation once, materialize it for training, and serve the same materialized values online. The mechanism differs; the principle is identical.

Point-in-time correctness

Time skew is the subtlest of the four and produces the most spectacular metric inflation.

The setup: you are predicting whether a signup will convert within thirty days. You join a customer features table that includes total_orders_ever. Offline accuracy is excellent. In production the model is barely better than guessing.

The reason is that total_orders_ever was computed when you built the dataset — today — and includes orders placed after the prediction moment. For a customer who signed up in March, the feature includes their April and May orders. The model learned that customers with many orders convert, which is true and completely unusable, because at prediction time that number is always zero or one.

The correct join reconstructs what was knowable at the prediction timestamp:

-- Every aggregate is bounded by the prediction timestamp.
-- The model can only see what existed when the decision was made.
SELECT
  p.customer_id,
  p.prediction_ts,
  COUNT(o.order_id)  AS orders_before_ts,
  MAX(o.created_at)  AS last_order_before_ts
FROM predictions p
LEFT JOIN orders o
  ON  o.customer_id = p.customer_id
  AND o.created_at  < p.prediction_ts   -- the entire point of the query
GROUP BY p.customer_id, p.prediction_ts

Two habits catch this class before it reaches a dashboard.

Be suspicious of large improvements. A single feature that adds ten points of accuracy is more likely a leak than an insight. The check costs five minutes: ask whether that value could have been known at prediction time.

Prefer temporal splits. Random splits hide leakage because the same time period appears in both train and test. Split by time — train on everything before a cutoff, test on everything after — and most leaks become visible as a gap between validation and test performance.

Mutable dimension tables deserve special suspicion. A customer's account_tier column shows today's tier, not their tier at prediction time. Slowly changing dimensions with validity ranges are the correct representation, and "we just join the current table" is the most common source of quiet time skew in production feature pipelines.

Detect skew by logging what you served

Prevention is better, but you also need detection, and detection requires one specific piece of instrumentation: log the feature vector that was actually served, not the raw request.

This is the single highest-value logging decision in a production ML system, and it is frequently skipped because the raw request feels like enough information. It is not. The raw request lets you recompute what the features should have been. Only the served vector tells you what they were.

With served vectors logged, three checks become possible:

Distribution comparison. Compare the served feature distribution against the training distribution, per feature. A feature whose mean has shifted by several standard deviations, or whose null rate went from two percent to forty, is a defect signal — usually an upstream schema change rather than a change in the world.

Replay. Take a day of logged served vectors, run them through the training-time transform from the raw inputs, and compare. Any mismatch is code skew, located precisely. This is the definitive test, and it is only possible if you logged both.

Training on served features. The strongest form: build the next training set from logged served vectors joined to outcomes, rather than recomputing features from the warehouse. Skew becomes structurally impossible, because training and serving use literally the same values. It requires discipline about logging and storage, and it is worth it for high-stakes models.

Worked example: the fraud model that got worse on Tuesdays

A fraud model performs well overall but has an elevated false-negative rate that clusters oddly. Offline metrics are unchanged. The team walks the four origins.

Distribution skew? The input distribution is compared against training. Nothing has moved meaningfully. The model also did not degrade over time — the pattern was present from launch, once someone looked for it. That rules out the origin everyone assumed.

Source skew? Training reads the warehouse; serving reads a live merchant API. They compare a sample of records and find agreement on values — but the live API returns null for merchant_category when its upstream lookup times out, while the warehouse has that field backfilled. Null rate at serving is four percent; in training it is zero.

Code skew? The imputation for a null merchant_category in serving is the string "unknown". The training pipeline never produced "unknown", because it never saw a null. The model has no learned representation for that category and treats those transactions as an unfamiliar merchant type.

Time skew? Checked and clean, because the team split temporally from the start.

The clustering is explained by the upstream lookup timing out more often during a batch job that runs on Tuesday mornings. Two defects compounding: a source difference that produced nulls, and a serving-side imputation invented locally to handle them.

The fix has three parts, and only the first was obvious. Fitted imputation values move into the artifact so serving cannot invent a fallback. The served feature vector is logged so the null rate is visible on a dashboard. And a data validation check fails the request loudly when the null rate for a required feature exceeds a threshold, rather than silently substituting a value the model has never seen.

Common mistake

The most common mistake is equating skew with drift and building only drift monitoring. Drift detection watches the world change. It will not tell you that your serving path imputes zero where training imputed the mean, because both distributions can look stable while being different from each other.

The second mistake is logging raw requests instead of served features. It feels equivalent and is not: recomputing features from a raw request reproduces what the transform does today, which is the very thing under suspicion. Without the served vector, replay comparison is impossible and code skew stays invisible.

The correcting exercise: take one production model, pull a hundred logged served feature vectors, and recompute the same features from the raw inputs using the training pipeline. Every mismatch is skew, already in production, already affecting predictions. Teams that run this for the first time are usually surprised — and the surprises are almost always in null handling, timezones, and rounding.