Build a Defensible Machine-Learning Baseline

Start with the decision, not the algorithm

A model is useful only when its output improves a decision. Write the actor, decision, available information, prediction horizon, and cost of a wrong answer. “Predict churn” is incomplete; “help a retention team choose which active customers to contact in the next seven days” can be tested.

Define the non-ML baseline. It might be a fixed rule, the majority class, last week’s value, or a human workflow. Without that reference, a high-looking metric can hide a system that adds no practical value.

Build the split before the features

Separate training, validation, and test data using the way the system will encounter the future. Time-based problems usually need a time-based split. Multiple records from one user, patient, device, or document family should not leak across splits. Fit normalization, vocabulary, and imputation on training data only.

Record dataset identity, row counts, date range, label definition, exclusions, and the code that produced each split. A result that cannot be reproduced is not a baseline.

Choose metrics that reflect failure costs

Accuracy can be misleading when classes are imbalanced. Compare precision, recall, confusion matrix, calibration, and performance for important subgroups. Add a product metric: review workload, false-alarm cost, missed-case cost, or time saved. State the threshold rather than reporting only an aggregate score.

Practice: baseline evidence sheet

Choose one public or synthetic dataset. Write the decision, label, prediction time, split rule, leakage risks, non-ML baseline, model baseline, two technical metrics, and one product metric. Run the same script twice and confirm identical split counts and baseline results.

Release check

Do not tune a larger model until you can explain why the baseline fails, which examples matter, and how much improvement would change the real decision.