Monitoring for Data and Model Drift

The Silent Failure Problem

Your model performs great in testing: 92% accuracy on a held-out test set. You deploy it to production. Three months later, a data analyst notices that model predictions have drifted—the distribution of predicted probabilities has shifted, and user complaints are rising. You check the test set accuracy: still 92%. Why is the model failing?

The answer: your model's performance depends on an implicit assumption about input distributions. In the real world, that assumption breaks. The input data distribution has changed (data drift), or the relationship between inputs and labels has changed (concept drift), or both. Your model hasn't changed; the world has.

This lesson teaches you to detect drift before it silently kills your model. You'll learn the difference between data and concept drift, practical statistical tests to detect them, and what to do when you find them.

Three Types of Drift: Data Drift, Concept Drift, and Label Shift

1. Data drift (covariate shift): The input feature distribution changes, but the relationship between inputs and labels stays the same. The model was trained on one input distribution; production has a different distribution.

Example: Your fraud detection model is trained on transactions from users aged 20-50. In production, 50% of new users are aged 60+. The input distribution has shifted. The model still understands fraud patterns (given a feature vector, it knows if it's fraud), but it's seeing ages it wasn't trained on.

Impact: May cause poor performance if the model extrapolates badly. Less severe than concept drift.

2. Concept drift: The relationship between inputs and labels changes. The input distribution stays the same, but the label definition or real-world dynamics change.

Example: Your loan default model is trained on historical data (20% default rate). Then new regulations change lending practices, and the real default rate drops to 10%. You're still seeing similar customers, but the concept of "default" has shifted due to changed economic conditions, regulations, or customer behavior.

Impact: Severe. Retraining on old data won't help; you need new labeled data reflecting the new concept.

3. Label shift (class imbalance drift): The distribution of labels changes, but the relationship between features and labels is unchanged. This is a special case of concept drift.

Example: A rare disease detection model sees more diseased patients in summer (seasonal pattern). The input features stay similar; only the proportion of positive labels changes.

Impact: Can fool your model into wrong decisions if not handled properly. Auto-balancing models (that assume equal class frequencies) fail.

# How to detect which type of drift occurred

import numpy as np
from scipy.stats import ks_2samp

# Scenario: Your model's performance dropped
# Which type of drift caused it?

# 1. Check if input distributions changed (data drift indicator)
ks_stat_X, p_value_X = ks_2samp(training_features, production_features)

# 2. Check if label distribution changed (label shift indicator)
original_positive_rate = (y_train == 1).mean()
production_positive_rate = (y_prod == 1).mean()

# 3. Check if predictions changed (concept drift indicator)
original_prediction_dist = model_original.predict_proba(X_validation)[:, 1]
current_prediction_dist = model_current.predict_proba(X_validation)[:, 1]
ks_stat_pred, p_value_pred = ks_2samp(original_prediction_dist, current_prediction_dist)

print(f"Data drift (p={p_value_X:.3f}): {p_value_X < 0.05}")
print(f"Label shift: {abs(original_positive_rate - production_positive_rate) > 0.05}")
print(f"Prediction drift (p={p_value_pred:.3f}): {p_value_pred < 0.05}")

# Diagnosis:
# - Only data drift: Model extrapolation may be poor; retrain may help
# - Only label shift: Model learned P(Y|X) correctly; you may need to recalibrate decision threshold
# - Only concept drift: Relationship changed; retraining essential
# - Multiple types: Investigate each separately

All three are dangerous, but they require different mitigation strategies:

• Data drift: Retrain on new data; or use domain adaptation techniques
• Concept drift: Retrain on new labeled data; understand root cause (regulations? market shift?)
• Label shift: Recalibrate decision threshold; or retrain with new label distribution

Detecting Data Drift: Statistical Tests

Kolmogorov-Smirnov Test

The KS test compares two distributions and outputs a p-value indicating if they're significantly different. For a feature in your model:

import numpy as np
from scipy.stats import ks_2samp

# Historical training data
training_feature = np.random.normal(loc=100, scale=15, size=5000)

# Production data from the last week
production_feature = np.random.normal(loc=105, scale=15, size=1000)

# KS test: null hypothesis is that distributions are the same
ks_statistic, p_value = ks_2samp(training_feature, production_feature)

print(f"KS statistic: {ks_statistic:.4f}")
print(f"p-value: {p_value:.6f}")

if p_value < 0.05:
    print("ALERT: Feature distribution has shifted significantly (p < 0.05)")
else:
    print("No significant drift detected")

The KS statistic ranges from 0 to 1 (0 = identical distributions, 1 = completely different). A p-value < 0.05 indicates a statistically significant difference.

Strengths:

• Simple, no hyperparameters
• Works on any distribution (univariate)
• Well-established statistical foundation

Limitations:

• Only tests individual features; doesn't capture multivariate shifts
• Sensitive to sample size (small differences matter less with small samples)
• Not ideal for discrete/categorical features

Population Stability Index (PSI)

PSI measures how much a distribution has shifted relative to a baseline. It's commonly used in credit risk modeling:

def population_stability_index(baseline, current, bins=10):
    """
    Calculate PSI: measure how much current distribution
    has shifted from baseline.

    PSI = sum((current_percent - baseline_percent) * ln(current_percent / baseline_percent))

    Interpretation:
    - PSI < 0.1: No significant population change
    - 0.1 <= PSI < 0.25: Small population change
    - PSI >= 0.25: Significant population change
    """
    # Bin the baseline data
    breakpoints = np.percentile(baseline, np.linspace(0, 100, bins + 1))
    breakpoints[0] = -np.inf
    breakpoints[-1] = np.inf

    # Histogram baseline
    baseline_counts, _ = np.histogram(baseline, bins=breakpoints)
    baseline_pct = baseline_counts / len(baseline)

    # Histogram current
    current_counts, _ = np.histogram(current, bins=breakpoints)
    current_pct = current_counts / len(current)

    # Avoid log(0)
    baseline_pct = np.where(baseline_pct == 0, 0.0001, baseline_pct)
    current_pct = np.where(current_pct == 0, 0.0001, current_pct)

    psi = np.sum((current_pct - baseline_pct) * np.log(current_pct / baseline_pct))
    return psi

# Example usage
baseline_age = np.random.normal(loc=35, scale=12, size=10000)
current_age = np.random.normal(loc=38, scale=12, size=1000)

psi = population_stability_index(baseline_age, current_age)
print(f"PSI: {psi:.4f}")

if psi < 0.1:
    print("No significant drift")
elif psi < 0.25:
    print("Small drift detected; monitor but OK to continue")
else:
    print("ALERT: Significant drift; consider retraining")

PSI is useful for tracking drift over time. Run it weekly or daily, compare current data to a fixed baseline (training data) or a rolling baseline (data from the past N days).

Multivariate Drift Detection

Both KS and PSI work on individual features. For high-dimensional data, you need multivariate tests. A simple approach: run KS/PSI on all features and alert if any exceed the threshold. More sophisticated: use dimensionality reduction (PCA) and test the principal components, or use a learned drift detector.

# Monitor multiple features
features = ['age', 'income', 'credit_score', 'account_length']
drift_alerts = {}

for feature in features:
    ks_stat, p_val = ks_2samp(training_data[feature], production_data[feature])
    drift_alerts[feature] = p_val < 0.05

if any(drift_alerts.values()):
    print("Data drift detected in features:", [f for f, d in drift_alerts.items() if d])

Detecting Concept Drift: Model Performance Monitoring

Data drift is detectable without labels. Concept drift requires ground truth. You need a way to get labels on a sample of production predictions, then track metrics over time.

In some domains, labels arrive naturally:

• Loan defaults: you get labels months after approval
• Click predictions: you know if a user clicked within minutes
• Fraud detection: fraud is confirmed over days

In others, you must actively collect labels (survey users, hire human raters).

import pandas as pd

# Track model performance metrics daily
# Assuming you get labels on 100-500 samples per day

daily_metrics = []

for day in date_range('2026-01-01', '2026-07-24'):
    # Get predictions and labels for that day
    predictions = get_predictions_for_day(day)
    labels = get_labels_for_day(day)

    # Calculate metrics
    accuracy = (predictions == labels).mean()
    precision = (predictions[predictions == 1] == labels[predictions == 1]).mean()
    recall = (predictions[labels == 1] == labels).sum() / (labels == 1).sum()

    daily_metrics.append({
        'date': day,
        'accuracy': accuracy,
        'precision': precision,
        'recall': recall,
        'sample_size': len(labels)
    })

df = pd.DataFrame(daily_metrics)

# Alert if accuracy drops > 5% from baseline
baseline_accuracy = df['accuracy'].iloc[:30].mean()  # First 30 days
current_accuracy = df['accuracy'].iloc[-7:].mean()   # Last 7 days

if current_accuracy < baseline_accuracy - 0.05:
    print(f"ALERT: Accuracy dropped from {baseline_accuracy:.1%} to {current_accuracy:.1%}")

Visualization matters: plot your metrics over time and look for trends.

Operational Response to Drift

Detecting drift is half the battle. You must act on it quickly and decisively. Here's a realistic operational playbook:

Stage 1: Alert and Investigate (Automated)

• Statistical test detects drift (KS, PSI, performance drop)
• System automatically logs which features/metrics triggered alert
• Alert routed to on-call data engineer with severity level (green/yellow/red)
• Dashboard shows drift magnitude and time of onset

Stage 2: Triage and Diagnosis (Human, < 1 hour)

• Data analyst investigates root cause: seasonal effect? data quality issue? production bug?
• Check: is this data drift, concept drift, or label shift?
• Review recent deployments, data pipeline changes, business events
• Determine if action is urgent (red alert, impacting customers) or can wait (yellow, minor drift)

Stage 3: Decide on Action

# Decision tree based on drift diagnosis

def decide_mitigation(drift_type, magnitude, days_since_training):
    if drift_type == 'data_drift':
        if magnitude < 0.1:  # Small drift
            return 'monitor'  # Watch but don't act yet
        else:
            return 'retrain_on_recent_data'  # Adapt model to new distribution

    elif drift_type == 'concept_drift':
        if magnitude < 0.05:  # Small concept shift
            return 'investigate_root_cause'  # May be temporary
        else:
            return 'urgent_retrain_and_review'  # Relationship changed; needs investigation

    elif drift_type == 'label_shift':
        return 'recalibrate_threshold'  # Adjust decision threshold, may not need retraining

    # Also trigger retraining if it's been > 30 days (preventive schedule)
    if days_since_training > 30:
        return 'scheduled_retrain'

    return 'no_action'

Stage 4: Mitigation (Automated, < 24 hours)

For data drift:

• Retrain on recent 90 days of data (gives model time to adapt)
• Validate on held-out recent data
• Deploy via canary if validation passes

For concept drift:

• Investigate root cause (business change? market shift?)
• Decide: is this permanent or temporary?
• If permanent, retrain + get stakeholder buy-in on any accuracy tradeoff
• If temporary, may not need action (e.g., seasonal effect goes away)

For label shift:

• Recalibrate decision threshold using new label distribution
• May not need retraining, just adjustment
• Monitor for concept drift separately

# Example: intelligent retraining pipeline
class DriftResponse:
    def __init__(self, current_model, baseline_accuracy):
        self.current_model = current_model
        self.baseline_accuracy = baseline_accuracy

    def respond_to_drift(self, drift_diagnosis):
        drift_type, magnitude = drift_diagnosis

        if drift_type == 'label_shift':
            # Simple fix: recalibrate threshold
            new_threshold = find_optimal_threshold_for_new_distribution()
            return {'action': 'recalibrate', 'threshold': new_threshold}

        elif drift_type == 'data_drift' or drift_type == 'concept_drift':
            # Retrain required
            X_recent, y_recent = get_recent_labeled_data(days=90)
            model_new = retrain(X_recent, y_recent)

            # Validate
            acc_new = evaluate(model_new, validation_set)
            acc_old = evaluate(self.current_model, validation_set)
            accuracy_loss = self.baseline_accuracy - acc_new

            if accuracy_loss < 0.02:  # Allow 2% loss
                return {
                    'action': 'deploy_retrained',
                    'model': model_new,
                    'deployment': 'canary'  # Start with 5% traffic
                }
            else:
                return {
                    'action': 'human_review',
                    'reason': f'Accuracy loss {accuracy_loss:.1%} exceeds tolerance',
                    'model': model_new
                }

    def validate_safety(self, new_model):
        # Before deployment: check fairness, latency, etc.
        # (See lesson 24 for fairness checks)
        pass

Stage 5: Deployment and Monitoring

• Deploy via canary (5% traffic for 1 hour, then 25%, then 100%)
• Monitor p95 latency, error rates, prediction distribution for deviations
• If metrics degrade, rollback to previous model automatically
• Keep human in loop for major changes

Stage 6: Prevention (Ongoing)

• Add detected drift as a regression test (never let this drift happen undetected again)
• Update data validation pipeline to catch similar drifts earlier
• Establish retraining schedule (weekly, monthly) as preventive measure
• Build model staleness dashboard (days since retraining)

Quantifying Drift: Setting Thresholds

Detecting drift is one thing; deciding when to act is another. Set thresholds that trigger investigation or action:

For data drift (KS/PSI):

• Green (OK): KS p-value > 0.05 or PSI < 0.1
• Yellow (investigate): KS p-value 0.01-0.05 or PSI 0.1-0.25
• Red (act): KS p-value < 0.01 or PSI > 0.25

For model performance:

• Green (OK): Accuracy within 2% of baseline
• Yellow (investigate): Accuracy 2-5% below baseline
• Red (act): Accuracy > 5% below baseline

For prediction distribution:

• Green (OK): Prediction distribution stable (KS p > 0.05)
• Yellow (investigate): Slight shift in distribution
• Red (act): Major shift (e.g., binary classifier predicting 90% positive when baseline was 50%)

These thresholds are illustrative; tune based on your domain. In lending, even 1% accuracy drop might be critical (high cost of mistakes). In recommendation systems, 5% drop might be acceptable.

Implementing Drift Monitoring in Production

import pandas as pd
import numpy as np
from scipy.stats import ks_2samp
from datetime import datetime, timedelta

class DriftMonitor:
    def __init__(self, baseline_data, feature_names, alert_threshold_pvalue=0.05):
        self.baseline_data = baseline_data
        self.feature_names = feature_names
        self.alert_threshold = alert_threshold_pvalue
        self.drift_history = []

    def check_drift(self, current_data):
        """
        Check for data drift in current batch vs. baseline.
        Returns: dict with p-values for each feature and overall alert status.
        """
        results = {
            'timestamp': datetime.now(),
            'features': {},
            'alert': False
        }

        for feature in self.feature_names:
            baseline_vals = self.baseline_data[feature]
            current_vals = current_data[feature]

            ks_stat, p_value = ks_2samp(baseline_vals, current_vals)

            results['features'][feature] = {
                'ks_statistic': ks_stat,
                'p_value': p_value,
                'alert': p_value < self.alert_threshold
            }

            if p_value < self.alert_threshold:
                results['alert'] = True

        self.drift_history.append(results)
        return results

    def summarize_drift(self, days=7):
        """
        Summarize drift over the past N days.
        Useful for trend detection.
        """
        cutoff = datetime.now() - timedelta(days=days)
        recent = [d for d in self.drift_history if d['timestamp'] > cutoff]

        summary = {}
        for feature in self.feature_names:
            alerts = sum(1 for d in recent if d['features'][feature]['alert'])
            summary[feature] = {
                'num_alerts': alerts,
                'alert_rate': alerts / len(recent) if recent else 0
            }

        return summary

# Usage
baseline_data = get_training_data()
monitor = DriftMonitor(baseline_data, feature_names=['age', 'income', 'score'])

# Each day, check for drift
current_batch = get_production_data_from_today()
drift_report = monitor.check_drift(current_batch)

if drift_report['alert']:
    alert_team("Data drift detected")
    for feature, details in drift_report['features'].items():
        if details['alert']:
            print(f"Feature {feature}: p-value = {details['p_value']:.6f}")

# Weekly summary
weekly_summary = monitor.summarize_drift(days=7)
print(weekly_summary)

Drift Detection Method Comparison

Different statistical tests work for different scenarios:

| Test | Use Case | Pros | Cons | Threshold | |------|----------|------|------|-----------| | Kolmogorov-Smirnov (KS) | Any univariate distribution | Simple, distribution-free, well-established | Only tests one feature at a time | p < 0.05 | | Population Stability Index (PSI) | Binned data, credit/finance | Business-friendly interpretation | Requires binning decisions | PSI < 0.1 (green), < 0.25 (yellow), >= 0.25 (red) | | Wasserstein distance | High-dimensional data | Better for multivariate, robust | Slower computation | Custom threshold | | Maximum Mean Discrepancy (MMD) | Complex distributions | Powerful for multivariate | Requires kernel selection | p < 0.05 | | Label distribution change | Concept drift (label shift) | Detects behavior shift | Requires labels (delayed) | > 5% change | | Model performance drop | Concept drift (relationship shift) | Direct impact on business | Requires labeled validation data | > 2-5% accuracy drop |

Recommendation: Start with KS test on features + model performance tracking. Add PSI if in finance/lending domain. Use multivariate tests (MMD) only if you have budget for ML infrastructure.

Monitoring vs. Retraining Strategy

| Scenario | Detection Latency | Action | Frequency | Cost | |----------|-------------------|--------|-----------|------| | No monitoring | Days-weeks (customer complaints) | Reactive retrain | Manual | High (downtime, lost revenue) | | Weekly KS test | 1 week | Check if drifted | Automated | Low-Medium | | Daily KS test | 1 day | Immediate investigation | Automated | Low | | Hourly performance check | 1 hour | Auto-retrain if needed | Automated + scheduled | Medium | | Real-time streaming + ML | Minutes | Immediate action | Fully automated | High (infra) |

Best practice: Daily KS test + weekly scheduled retraining + on-demand retraining if drift detected.

Monitoring Checklist

Set up these checks in your production pipeline:

| Check | Frequency | Action if Alert | |-------|-----------|-----------------| | Data drift (KS test or PSI) | Daily | Investigate root cause; check for data quality issues | | Model accuracy (labeled sample) | Weekly | Retrain if significant drop; investigate root cause | | Prediction distribution | Daily | Check for mode collapse or feature shift; verify model output | | Model latency | Hourly | Scale resources or roll back to previous version | | Feature value ranges | Daily | Alert if out-of-range values appear; check data pipeline | | Label distribution | Weekly | Investigate if user behavior or world has changed |

For high-stakes applications, monitor daily. For lower-stakes applications, weekly may be sufficient. The key: monitor from day 1, don't wait for catastrophic failure.

Common Mistake: Waiting for a Catastrophic Failure

Many teams don't set up proactive drift monitoring. They wait for user complaints, customer churn, or regulatory action. By then, your model has been degrading silently for months.

Deploy drift monitoring in week 1 of production, not month 6. It takes hours to set up and saves months of silent failure.

Also, don't ignore "small" drift. A 5% performance drop seems tolerable, but over a quarter it compounds: 95% → 90.3% → 85.8% → 81.5%. Catch drift early and retrain often.

# Right: Monitor continuously
while True:
    drift_detected = run_drift_checks()
    if drift_detected:
        log_alert("Drift detected")
        trigger_investigation()

# Wrong: Wait for catastrophe
# (don't do this!)

Drift is inevitable in production. Your job is to detect it quickly and respond decisively.