Scikit-learn: Classical Machine Learning in Python

Scikit-learn covers the classical toolkit: linear and logistic regression, trees, random forests, gradient boosting, support vector machines, clustering, dimensionality reduction, and the preprocessing that feeds them. For structured tabular data — which is most business data — these methods train in seconds, explain themselves, and frequently match or beat a neural network on the same problem.

That last point is not nostalgia. On tabular data with a few thousand to a few million rows, gradient-boosted trees remain extremely competitive, and the deep learning alternative usually costs more to build, more to serve, and more to explain to whoever has to sign off on the decision.

One interface, every algorithm

The defining design choice is that nearly everything implements the same small interface. Models have fit and predict. Transformers have fit and transform. Anything that scores has score. Hyperparameters are constructor arguments, readable and settable.

The practical payoff is that comparing algorithms is a loop rather than a rewrite:

from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

for model in [LogisticRegression(max_iter=1000), RandomForestClassifier(), HistGradientBoostingClassifier()]:
    scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
    print(f"{model.__class__.__name__}: {scores.mean():.3f} +/- {scores.std():.3f}")

Always start with the simplest model as a baseline. A logistic regression that scores within a couple of points of a tuned ensemble is usually the right thing to ship, because it trains instantly, explains its coefficients, and will not surprise anyone in six months. The baseline also tells you whether the problem is learnable at all — if nothing beats predicting the majority class, more modelling will not help, and the answer lies in the features or the data.

The pipeline is a correctness feature

Fitting a scaler on all your data and then splitting into train and test is the most common serious mistake in applied machine learning. The scaler has seen the test set's statistics, so your validation score is optimistic, and the size of the optimism is unpredictable. The same applies to imputation, encoding, and feature selection.

Pipeline exists to make this structurally impossible. Everything inside it is fit only on the training fold during cross-validation:

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

preprocess = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer()), ("scale", StandardScaler())]), numeric_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_cols),
])

model = Pipeline([("prep", preprocess), ("clf", HistGradientBoostingClassifier())])
scores = cross_val_score(model, X, y, cv=5)  # every fold refits the preprocessing

handle_unknown="ignore" in that snippet is not incidental. Production will send you a category the training data never contained, and the default behaviour is to raise.

A pipeline also solves deployment: the fitted object contains the preprocessing, so serving cannot drift from training. Reimplementing the preprocessing in the serving code is how training and serving quietly diverge.

Validate the way the model will be used

The default k-fold split assumes rows are independent and interchangeable. Frequently they are not, and when they are not, cross-validation reports a score the model will never achieve.

Time series: never shuffle. Use a forward-chaining split, because in production you predict the future from the past and a shuffled split lets the model see it.

Grouped data: multiple rows per customer, patient, or session must not be split across folds, or the model recognises the entity rather than learning the pattern.

Imbalanced classes: stratify, and stop looking at accuracy. Use precision, recall, and the confusion matrix, and pick the decision threshold deliberately rather than accepting 0.5 — the threshold is a business decision about the relative cost of the two error types, not a modelling detail.

For hyperparameters, randomised search usually finds a good region faster than an exhaustive grid, and any tuning must happen inside the cross-validation loop or you have leaked again, this time through the selection.

Explaining what the model learned

One of the underrated reasons to stay with classical methods is that you can answer "why did it decide that?" without a research project.

Linear and logistic regression give you coefficients directly, though they are only comparable across features if the features were scaled. Tree ensembles expose feature importances, with the caveat that the default impurity-based version is biased toward high-cardinality features — permutation importance, which measures how much the score drops when you shuffle one column, is slower and more trustworthy.

Partial dependence plots show how the prediction moves as one feature varies while the others are held at their observed distribution, which is usually what a stakeholder means when they ask about the effect of a variable.

Two cautions worth carrying into any such conversation. Importance is not causation: a feature can be important because it is a proxy for something else entirely, and acting on it will not change the outcome. And a feature that is unexpectedly dominant is more often a leak than an insight — if one column predicts the target almost perfectly, check whether it is recorded after the event you are trying to predict.

Where it stops

Scikit-learn is not for neural networks, does not use GPUs, and does not target images, audio, or long text. Its role in a modern stack is the baseline, the tabular work, and the preprocessing and evaluation utilities that pair with a deep learning framework when one is genuinely needed. Most teams use it alongside PyTorch rather than instead of it.

Common mistakes

Preprocessing outside the pipeline. Leaks test statistics into training and inflates every number you report.

Skipping the simple baseline. Without it you cannot tell whether the complex model earned its complexity.

Shuffling time series. Produces a score that cannot be reproduced in production.

Accepting the 0.5 threshold. It is a business decision, and the default is rarely the right one.