Build a Defensible Machine-Learning Baseline

Start with the decision, not the algorithm

A model is useful only when its output improves a decision. Before writing code or considering algorithms, write down the actor (who makes the decision?), decision (what action do they take?), available information (what data do they have at prediction time?), prediction horizon (how far into the future?), and cost of error (what breaks if we get it wrong?).

"Predict churn" is incomplete and unmeasurable. "Help a retention team choose which active customers to contact in the next seven days to reduce churn" is testable: you can measure whether the team's decision improves, whether predictions reach the right window, and whether the cost of contacting someone is justified by prevented churn.

Define a non-ML baseline. This might be a fixed rule ("contact customers whose last purchase was >90 days ago"), the majority class ("always predict 'no churn'"), last value ("use yesterday's churn rate"), or a human workflow ("support agents flag high-risk customers by intuition"). Without this reference point, a high-looking metric (like 92% accuracy) can hide a system that adds no practical value — if the baseline already achieves 91%, your complex model bought only 1 percentage point at the cost of latency, interpretability, and technical debt.

Build the split before the features

Separate training, validation, and test data using the way the system will encounter the future. Time-based problems usually need a time-based split: if you are predicting churn next month, train on data from months 1–8, validate on month 9, and test on month 10. Never put historical data in training and future data in test — that is leakage, and it will hide generalization failure.

Multiple records from one user, patient, device, or document family must not leak across splits. If a customer's 50 transactions are scattered randomly across train and test, the model can memorize user-level patterns instead of learning generalizable features.

Fit normalization, vocabulary, and imputation on training data only, then apply those transformations to validation and test. This mimics what happens in production: you fit once on historical data and reuse those parameters for all future predictions.

Record dataset identity before writing any model code: row counts, date range, label definition, exclusion criteria, and the exact code that produced each split. A result that cannot be reproduced is not a baseline — you will not trust it, and the next person to work on the problem will waste time rediscovering the split logic.

Example: reproducible train/validation/test split

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler

# Load raw data with a timestamp column
raw_df = pd.read_csv("customer_data.csv")
raw_df["timestamp"] = pd.to_datetime(raw_df["timestamp"])

# Sort by time to ensure reproducibility
raw_df = raw_df.sort_values("timestamp").reset_index(drop=True)

# Time-based split: train on first 60%, validate on next 20%, test on last 20%
n = len(raw_df)
train_end = int(0.6 * n)
val_end = int(0.8 * n)

train_df = raw_df.iloc[:train_end].copy()
val_df = raw_df.iloc[train_end:val_end].copy()
test_df = raw_df.iloc[val_end:].copy()

print(f"Train: {len(train_df)} rows, {train_df['timestamp'].min()} to {train_df['timestamp'].max()}")
print(f"Val: {len(val_df)} rows, {val_df['timestamp'].min()} to {val_df['timestamp'].max()}")
print(f"Test: {len(test_df)} rows, {test_df['timestamp'].min()} to {test_df['timestamp'].max()}")

# Compute normalization parameters on training data only
scaler = StandardScaler()
numeric_cols = ["age", "account_balance", "months_active"]
scaler.fit(train_df[numeric_cols])

# Apply the same scaler to validation and test
train_df[numeric_cols] = scaler.transform(train_df[numeric_cols])
val_df[numeric_cols] = scaler.transform(val_df[numeric_cols])
test_df[numeric_cols] = scaler.transform(test_df[numeric_cols])

# Extract features and labels for the baseline
X_train, y_train = train_df.drop("churn", axis=1), train_df["churn"]
X_val, y_val = val_df.drop("churn", axis=1), val_df["churn"]
X_test, y_test = test_df.drop("churn", axis=1), test_df["churn"]

print(f"Class balance (train): {y_train.value_counts().to_dict()}")
print(f"Class balance (val): {y_val.value_counts().to_dict()}")
print(f"Class balance (test): {y_test.value_counts().to_dict()}")

This code ensures that the split is deterministic (always the same result), that normalization is fit only on training data, and that all splits are logged for reproducibility.

Define a non-ML baseline

A baseline is a model or rule that requires no machine learning. It serves as a sanity check: if your trained model cannot beat the baseline by a meaningful margin, the added complexity is not justified.

Common baselines include:

• Majority class: Always predict the most common label. For a churn dataset with 90% non-churners, predict "no churn" for everyone — you get 90% accuracy.
• Last observed value: For time series, predict today's value as equal to yesterday's. For a temperature forecast, this is a strong baseline.
• Rule-based heuristic: "Flag customers who have not purchased in 180 days." This is interpretable, fast, and often beats a first model.
• Per-group average: For each user segment, compute the mean target value and use that as the prediction.

Compute the baseline's metrics on your validation set using the same metrics you will use to evaluate the learned model. This is critical: if you report only accuracy for the baseline but precision/recall for your model, you are making incomparable claims.

Example: baseline computation

# Majority class baseline
train_churn_rate = y_train.mean()
majority_pred = np.round(train_churn_rate)
baseline_accuracy = (majority_pred == y_val).mean()
print(f"Majority-class baseline accuracy on validation: {baseline_accuracy:.3f}")

# Rule-based baseline: flag if last purchase > 180 days ago
baseline_rule_pred = (val_df["days_since_purchase"] > 180).astype(int)
baseline_rule_accuracy = (baseline_rule_pred == y_val).mean()
print(f"Rule-based baseline accuracy on validation: {baseline_rule_accuracy:.3f}")

# Compare: if your trained model achieves 85% accuracy and the rule achieves 83%,
# the 2 percentage point gain may not justify production complexity.

Choose metrics that reflect failure costs

Accuracy can be misleading when classes are imbalanced. With 98% non-churners, a model that predicts "never churn" achieves 98% accuracy but catches zero churners.

Instead, report multiple metrics that expose the cost of different errors:

• Precision: Of the customers we predicted would churn, what fraction actually churned? (Precision = TP / (TP + FP))
• Recall: Of the customers who actually churned, what fraction did we catch? (Recall = TP / (TP + FN))
• Confusion matrix: A 2×2 table showing true positives, false positives, true negatives, and false negatives — it tells the whole story.
• Calibration: If we predict 30% probability for a customer, do ~30% of those customers actually churn? (Miscalibrated confidences are dangerous.)
• Per-subgroup performance: Does accuracy vary widely by age, region, or customer tenure? If so, the model is unfairly biased.

Add a product metric: review workload (how many customers do we contact?), false-alarm cost (how much do false positives cost?), missed-case cost (how much does missing a churner cost?), or time saved. A metric that reflects business impact is more useful than a technical one.

Example: multi-metric evaluation

from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score
from sklearn.linear_model import LogisticRegression

# Fit a simple learned model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

# Predict on validation set
y_val_pred = model.predict(X_val)
y_val_pred_proba = model.predict_proba(X_val)[:, 1]

# Compute technical metrics
cm = confusion_matrix(y_val, y_val_pred)
tn, fp, fn, tp = cm.ravel()

accuracy = (tp + tn) / (tp + tn + fp + fn)
precision = precision_score(y_val, y_val_pred)
recall = recall_score(y_val, y_val_pred)
f1 = f1_score(y_val, y_val_pred)

print(f"Confusion matrix:\n{cm}")
print(f"Accuracy: {accuracy:.3f}")
print(f"Precision: {precision:.3f} (of predicted churners, {precision:.1%} actually churned)")
print(f"Recall: {recall:.3f} (of actual churners, we caught {recall:.1%})")
print(f"F1 score: {f1:.3f}")

# Product metric: if contacting a customer costs $5 and losing a churner costs $100,
# what is the net value of this model?
contact_cost = 5
churn_cost = 100
n_predicted_positive = (y_val_pred == 1).sum()
value = tp * (churn_cost - contact_cost) - fp * contact_cost
print(f"Net value (if tp prevents churn): ${value} across {n_predicted_positive} contacts")

Practice: baseline evidence sheet

Choose one public or synthetic dataset (e.g., UCI ML Repository, Kaggle, or scikit-learn's built-in datasets). Write the baseline evidence sheet — a document capturing:

1. Actor & decision: Who uses this model, and what action do they take?
2. Label definition: How is the target variable computed? (e.g., "churn = no purchase for 90 days after the prediction date")
3. Prediction time: When is the model applied? (e.g., "at the start of each month")
4. Split rule: How are train/validation/test separated? (e.g., "time-based split: first 60% of rows by date, next 20%, last 20%")
5. Leakage risks: What could accidentally let the model cheat? (e.g., "user IDs must not appear in both train and test")
6. Non-ML baseline: What simple rule is your floor? (e.g., "majority class = 88% accuracy")
7. Learned baseline: A simple model (logistic regression, decision tree) on your features.
8. Two technical metrics: E.g., precision and recall, or F1 and AUC.
9. One product metric: E.g., net value, alert rate, or cost per caught case.

Run the same script twice and confirm identical split counts and baseline results. If the results differ, your code has randomness you did not control — fix it.

Example: baseline evidence checklist

# Minimal baseline evidence sheet for a public dataset
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score

# Load data
data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = pd.Series(data.target, name="diagnosis")

# Record dataset identity
print("=" * 60)
print("BASELINE EVIDENCE SHEET")
print("=" * 60)
print(f"Dataset: breast cancer diagnosis (UCI ML)")
print(f"Total samples: {len(X)}")
print(f"Features: {X.shape[1]}")
print(f"Label: {y.name} (1=malignant, 0=benign)")
print(f"Class distribution: {y.value_counts().to_dict()}")
print()

# Split: deterministic time-based would be ideal, but for this static dataset,
# use random_state for reproducibility
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42
)

print(f"Split (random, reproducible):")
print(f"  Train: {len(X_train)} ({len(X_train)/len(X):.1%})")
print(f"  Val: {len(X_val)} ({len(X_val)/len(X):.1%})")
print(f"  Test: {len(X_test)} ({len(X_test)/len(X):.1%})")
print()

# Non-ML baseline: majority class
baseline_pred = y_train.mode()[0]
baseline_acc = (baseline_pred == y_val).mean()
print(f"Non-ML baseline (majority class={baseline_pred}): {baseline_acc:.3f}")
print()

# Learned baseline: logistic regression
model = LogisticRegression(max_iter=1000, random_state=42)
model.fit(X_train, y_train)
y_val_pred = model.predict(X_val)

val_acc = accuracy_score(y_val, y_val_pred)
val_prec = precision_score(y_val, y_val_pred)
val_rec = recall_score(y_val, y_val_pred)

print(f"Learned baseline (logistic regression on val):")
print(f"  Accuracy: {val_acc:.3f}")
print(f"  Precision: {val_prec:.3f}")
print(f"  Recall: {val_rec:.3f}")
print()

# Final test evaluation
y_test_pred = model.predict(X_test)
test_acc = accuracy_score(y_test, y_test_pred)
test_prec = precision_score(y_test, y_test_pred)
test_rec = recall_score(y_test, y_test_pred)

print(f"Test set performance (held out until now):")
print(f"  Accuracy: {test_acc:.3f}")
print(f"  Precision: {test_prec:.3f}")
print(f"  Recall: {test_rec:.3f}")
print()

# Conclusion
print(f"Verdict: Model beats non-ML baseline by {(val_acc - baseline_acc):.1%}.")
print(f"Both precision and recall are strong; ready to investigate feature engineering.")

Running this script twice should yield identical numbers (thanks to random_state).

When baselines outperform complex models: a real case study

Telco Customer Churn Prediction (illustrative estimate from benchmark practices):

A telecom company built a churn prediction model. The business question: "Which customers are likely to cancel in the next 30 days so we can offer them a retention incentive?"

Non-ML baseline: A rule-based heuristic — "flag customers who have not used their service in 14+ days or have submitted support complaints in the last 7 days." This rule required zero training and could be deployed in SQL.

Learned model: A logistic regression model trained on 50 customer features (contract length, monthly charges, usage patterns, support tickets, etc.).

Results (on a holdout test set of ~10,000 customers):

| Model | Precision | Recall | F1 Score | Avg. Contact Cost | Net Lifetime Value Saved Per Contact | |-------|-----------|--------|----------|-------------------|---------------------------------------| | Rule-based baseline | 0.65 | 0.42 | 0.51 | ~$5 | $18 | | Logistic regression | 0.71 | 0.58 | 0.64 | ~$5 | $26 |

The logistic regression improved F1 by 0.13 points and net value by $8 per contact. On 100,000 customers per year, this is $800,000 in incremental value — easily justifying the engineering cost. But if the rule-based model had outperformed the learned model, the conclusion would have been different: stick with the rule and invest engineering effort elsewhere.

Release check

Do not tune a larger model, add more features, or invest in complex engineering until you can explain:

1. Why does the baseline fail? Look at examples the baseline gets wrong. Are they ambiguous, or is there a clear pattern? A strong baseline that fails on only a few examples suggests the problem is nearly solved; a weak baseline suggests you need different features or a fundamentally different approach.

2. Which examples matter? Are there subgroups where the baseline (or your first model) performs poorly? Focus on improving those. If the baseline has 90% accuracy overall but 60% accuracy on a specific customer segment, that segment is your target for improvement — feature engineering and model selection should focus there.

3. How much improvement justifies the cost? If a 2 percentage point lift in accuracy costs 5x latency, is it worth it for your use case? For a real-time fraud-detection system, 10 ms latency vs 50 ms latency is critical; for a batch email targeting system, 100 ms is irrelevant. Always measure cost in business terms (revenue impact, cost to deploy/maintain, latency impact on user experience) not just in accuracy points.

Baseline strategy table: when to use each approach

| Baseline Type | Use When | Example | Pros | Cons | |---|---|---|---|---| | Majority class | Imbalanced classification | "Always predict no churn" on 90% negative data | Fast, simple to compute | Unrealistic; hides poor recall on minority class | | Rule-based heuristic | Domain expertise exists | "Flag accounts > 90 days without purchase" | Interpretable, fast, often beats first ML model | Requires manual rule-crafting; brittle if rules miss edge cases | | Last value | Time series or seasonal data | "Predict next month's revenue = this month's" | Captures momentum | Fails if trend changes; misses seasonality | | Per-group average | Segmented predictions | "Predict churn rate = historical rate for that customer segment" | Incorporates domain structure | Sensitive to segment sizes; extrapolates poorly to new segments | | Learned baseline | No strong domain rules | "Simple logistic regression on raw features" | Data-driven; learns signal without complex tuning | Requires labeled data; less interpretable than rules |

Common leakage patterns and how to avoid them

Leakage is when information from the future or the label sneaks into training data, artificially inflating performance and guaranteeing failure in production. Here are five dangerous patterns:

| Leakage Pattern | Example | How to Detect | Fix | |---|---|---|---| | Future information in features | A feature computed from the label (e.g., "days until churn" computed from actual churn date) | Performance drops 50%+ on test data compared to validation | Remove features computed from the label; use only information available at prediction time | | User/entity leakage | Training: rows for user X; Test: different rows for user X. Model memorizes user patterns. | Per-user accuracy varies wildly; leakage-free split has lower overall accuracy | Split by entity (all rows of user X in test, none in train) | | Time-based leakage | Training on Jan–Mar; test on Jan–Mar with different random samples. Model has seen the time period. | Fold-based split has better scores than time-based split | Use time-based split: train on Jan–Feb, validate on Mar, test on Apr | | Preprocessing fit on full data | Scaler fit on train + test; then train test. Test data influences normalization parameters. | Subtle; shows up as overly optimistic validation metrics | Fit scaler/encoder on training data only; apply to test | | Data from after the decision | Predicting patient diagnosis, but training data includes test results ordered after admission | Diagnosis accuracy implausibly high (~95%+) | Timeline: use only data available at the decision point |

Release check

Do not tune a larger model, add more features, or invest in complex engineering until you can explain:

1. Why does the baseline fail? Look at examples the baseline gets wrong. Are they ambiguous, or is there a clear pattern? A strong baseline that fails on only a few examples suggests the problem is nearly solved; a weak baseline suggests you need different features or a fundamentally different approach.

2. Which examples matter? Are there subgroups where the baseline (or your first model) performs poorly? Focus on improving those. If the baseline has 90% accuracy overall but 60% accuracy on a specific customer segment, that segment is your target for improvement — feature engineering and model selection should focus there.

3. How much improvement justifies the cost? If a 2 percentage point lift in accuracy costs 5x latency, is it worth it for your use case? For a real-time fraud-detection system, 10 ms latency vs 50 ms latency is critical; for a batch email targeting system, 100 ms is irrelevant. Always measure cost in business terms (revenue impact, cost to deploy/maintain, latency impact on user experience) not just in accuracy points.

Common mistake: confusing baseline with ceiling

A weak baseline does not mean your problem is impossible — it means your baseline is weak. A strong baseline does not mean ML cannot help — it means any learned model must beat it by a meaningful margin. The baseline is a floor, not a ceiling: it sets a realistic lower bound on performance and forces you to ask "is this added complexity worth it?"