Classification and Decision Boundaries

Decision boundaries: where does the model say yes vs no?

A classifier divides the input space into regions, each assigned to a class. The decision boundary is the line (or surface in higher dimensions) separating these regions. The shape of the boundary depends on the model:

• Logistic regression: Linear boundary. Works well when classes are separable by a straight line.
• Decision trees: Axis-aligned rectangular boundaries. Can capture nonlinear patterns but are prone to overfitting.
• Random forests / gradient boosting: Smooth, irregular boundaries. Reduce overfitting by averaging many trees.
• Neural networks: Arbitrary curved boundaries. Can fit very complex patterns but require careful regularization.

The more complex the boundary, the more model capacity you are using. More capacity = better fit on training data but higher overfitting risk.

Logistic regression: a linear decision boundary

Logistic regression outputs a probability between 0 and 1 by combining two stages:

Stage 1: Linear combination (same as linear regression): z = w₀ + w₁·x₁ + w₂·x₂ + ... + wₙ·xₙ

Stage 2: Sigmoid function (squashes z to [0, 1]): P(class=1 | x; w) = sigmoid(z) = 1 / (1 + e^(-z))

The sigmoid function has these key properties:

• sigmoid(0) = 0.5 (decision boundary)
• sigmoid(∞) ≈ 1 (very confident positive)
• sigmoid(-∞) ≈ 0 (very confident negative)
• sigmoid'(z) = sigmoid(z) · (1 - sigmoid(z)) (used in gradient-based optimization)

Decision rule: Predict class=1 if P(class=1) ≥ threshold (default threshold = 0.5), else class=0.

The decision boundary is where P(class=1) = 0.5, which happens exactly where z = 0. This is a hyperplane in feature space — a straight line in 2D, a plane in 3D, etc. Points far from the boundary are classified confidently (P close to 0 or 1); points near the boundary are ambiguous (P close to 0.5).

Loss function for logistic regression (binary cross-entropy): J(w) = -(1/n) · Σᵢ[ y_i · log(P_i) + (1-y_i) · log(1-P_i) ]

where P_i is the predicted probability for sample i. This loss penalizes confident but wrong predictions (e.g., predicting P=0.95 when y=0) more heavily than uncertain predictions.

Example: logistic regression with 2 features

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

# Generate 2D classification data
np.random.seed(42)
X, y = make_classification(
    n_samples=300,
    n_features=2,
    n_informative=2,
    n_redundant=0,
    random_state=42
)

# Fit three classifiers
log_reg = LogisticRegression(random_state=42)
tree = DecisionTreeClassifier(max_depth=3, random_state=42)
forest = RandomForestClassifier(n_estimators=100, max_depth=3, random_state=42)

log_reg.fit(X, y)
tree.fit(X, y)
forest.fit(X, y)

# Create a mesh grid for decision boundary visualization
h = 0.02  # step size in the mesh
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                      np.arange(y_min, y_max, h))

# Plot decision boundaries
fig, axes = plt.subplots(1, 3, figsize=(15, 4))

models = [
    ("Logistic Regression (linear)", log_reg),
    ("Decision Tree (axis-aligned)", tree),
    ("Random Forest (smooth)", forest)
]

for ax, (title, model) in zip(axes, models):
    # Predict on mesh
    Z = model.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]
    Z = Z.reshape(xx.shape)

    # Plot contour lines (decision boundary at 0.5)
    ax.contourf(xx, yy, Z, levels=[0, 0.5, 1], colors=['lightblue', 'lightcoral'], alpha=0.6)
    ax.contour(xx, yy, Z, levels=[0.5], colors='black', linewidths=2)

    # Plot training points
    colors = ['blue', 'red']
    for i, color in enumerate(colors):
        idx = y == i
        ax.scatter(X[idx, 0], X[idx, 1], c=color, label=f'Class {i}', s=30, alpha=0.7)

    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    ax.set_xlabel("Feature 1")
    ax.set_ylabel("Feature 2")
    ax.set_title(title)
    ax.legend()

plt.tight_layout()
plt.show()

# Accuracy comparison
print(f"Training accuracy:")
print(f"  Logistic regression: {log_reg.score(X, y):.3f}")
print(f"  Decision tree:       {tree.score(X, y):.3f}")
print(f"  Random forest:       {forest.score(X, y):.3f}")

Classifier comparison table: boundary type, complexity, and bias-variance trade-off

| Aspect | Logistic Regression | Decision Tree | Random Forest | Neural Network | |--------|---------------|---------|---------|---------| | Boundary shape | Linear hyperplane | Axis-aligned rectangles | Irregular, smooth | Arbitrary curves | | Flexibility | Low (linear only) | Very high (can overfit) | High (robust via ensemble) | Very high | | Bias | High (may underfit) | Low (fits training data well) | Medium | Low | | Variance | Low | Very high (single tree overfits) | Low (averaging reduces variance) | High (requires regularization) | | Interpretability | High (see coefficients) | High (trace decision path) | Medium (feature importance) | Low (black-box) | | Training speed | Fast | Fast | Slower (many trees) | Slow (backpropagation) | | When to use | Linearly separable data, need interpretability | Nonlinear data, small dataset | Nonlinear data, want robustness | Complex patterns, lots of data | | Example use case | Loan approval (linear rules), medical risk scores | Small patient datasets | Fraud detection, e-commerce | Image classification |

Illustrative performance ranges (typical on medium-complexity problems):

• Logistic regression: 70–80% test accuracy
• Decision tree: 75–95% training accuracy, but only 65–75% test (overfitting)
• Random Forest: 80–90% test accuracy (reduced variance)
• Neural network: 85–95% test accuracy (if properly tuned and regularized)

Tree-based classifiers: nonlinear, axis-aligned boundaries

Decision trees recursively split the feature space along feature axes. Each split is of the form "if feature_j < threshold then go left else go right." This creates axis-aligned rectangular regions.

Advantage: Trees can capture nonlinear patterns and are interpretable (you can explain each decision path).

Disadvantage: A single deep tree overfits. Random forests and gradient boosting mitigate this by training many trees on different data subsets and averaging their predictions, creating smoother boundaries and better generalization.

Example: compare tree depth effects

from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
import matplotlib.pyplot as plt

# Generate data
np.random.seed(42)
X_train, y_train = make_classification(n_samples=200, n_features=2, n_informative=2,
                                        n_redundant=0, random_state=42)
X_test, y_test = make_classification(n_samples=200, n_features=2, n_informative=2,
                                      n_redundant=0, random_state=43)

# Fit trees of different depths
depths = [1, 2, 3, 5, 10, 20]
train_accs, test_accs = [], []

for depth in depths:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    tree.fit(X_train, y_train)
    train_accs.append(tree.score(X_train, y_train))
    test_accs.append(tree.score(X_test, y_test))

# Plot overfitting curve
plt.figure(figsize=(10, 5))
plt.plot(depths, train_accs, 'o-', label='Training accuracy', linewidth=2, markersize=8)
plt.plot(depths, test_accs, 's-', label='Test accuracy', linewidth=2, markersize=8)
plt.xlabel("Tree depth (model complexity)")
plt.ylabel("Accuracy")
plt.title("Overfitting: deeper trees fit training data better but generalize worse")
plt.legend()
plt.grid(True, alpha=0.3)
plt.xticks(depths)
plt.tight_layout()
plt.show()

print(f"Overfitting pattern:")
print(f"  Depth 1: train={train_accs[0]:.3f}, test={test_accs[0]:.3f}, gap={train_accs[0]-test_accs[0]:.3f}")
print(f"  Depth 3: train={train_accs[2]:.3f}, test={test_accs[2]:.3f}, gap={train_accs[2]-test_accs[2]:.3f}")
print(f"  Depth 20: train={train_accs[-1]:.3f}, test={test_accs[-1]:.3f}, gap={train_accs[-1]-test_accs[-1]:.3f}")

Reading a confusion matrix

A confusion matrix is a 2×2 table (for binary classification) showing which predictions were correct and which were wrong:

                 Predicted
                 Negative  Positive
Actual Negative   TN        FP
       Positive   FN        TP

• TP (true positive): Correctly predicted positive. "We said yes, it was yes."
• TN (true negative): Correctly predicted negative. "We said no, it was no."
• FP (false positive): Incorrectly predicted positive. "We said yes, it was no." (Type I error, false alarm)
• FN (false negative): Incorrectly predicted negative. "We said no, it was yes." (Type II error, missed case)

Derived metrics

• Accuracy = (TP + TN) / (TP + TN + FP + FN) — Fraction of correct predictions
• Precision = TP / (TP + FP) — Of positive predictions, how many were actually positive?
• Recall = TP / (TP + FN) — Of actual positives, how many did we catch?
• F1 score = 2 · (Precision · Recall) / (Precision + Recall) — Harmonic mean; balances precision and recall

Choose based on your use case:

• Precision matters when false positives are costly (e.g., spam filtering: flagging legitimate mail as spam is bad).
• Recall matters when false negatives are costly (e.g., disease screening: missing a case is worse than a false alarm).
• Accuracy matters when both errors are equally costly (e.g., balanced classification).

Example: computing and interpreting confusion matrix

from sklearn.metrics import confusion_matrix, classification_report, ConfusionMatrixDisplay
import matplotlib.pyplot as plt

# Train a classifier
np.random.seed(42)
X, y = make_classification(n_samples=500, n_features=10, n_informative=5,
                           n_redundant=2, n_classes=2, random_state=42)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Fit logistic regression
clf = LogisticRegression(max_iter=1000, random_state=42)
clf.fit(X_train, y_train)

# Predictions
y_pred = clf.predict(X_test)

# Confusion matrix
cm = confusion_matrix(y_test, y_pred)
tn, fp, fn, tp = cm.ravel()

print("=" * 60)
print("CONFUSION MATRIX AND METRICS")
print("=" * 60)
print(f"\nConfusion Matrix:")
print(f"  TN (correct negatives): {tn}")
print(f"  FP (false alarms):      {fp}")
print(f"  FN (missed cases):      {fn}")
print(f"  TP (correct positives): {tp}")
print()

accuracy = (tp + tn) / (tp + tn + fp + fn)
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * (precision * recall) / (precision + recall)

print(f"Accuracy:  {accuracy:.3f}  (overall correctness)")
print(f"Precision: {precision:.3f}  (of predicted positives, {precision*100:.1f}% are actually positive)")
print(f"Recall:    {recall:.3f}   (of actual positives, we caught {recall*100:.1f}%)")
print(f"F1 score:  {f1:.3f}   (balance of precision and recall)")
print()

# Full classification report
print("Classification Report (scikit-learn):")
print(classification_report(y_test, y_pred))

# Visualize confusion matrix
fig, ax = plt.subplots(figsize=(8, 6))
ConfusionMatrixDisplay(cm, display_labels=['Negative', 'Positive']).plot(ax=ax, cmap='Blues')
plt.title("Confusion Matrix")
plt.tight_layout()
plt.show()

Decision threshold and probability calibration

Logistic regression outputs probabilities. By default, predictions are made at a 0.5 threshold: if P(class=1) ≥ 0.5, predict positive; otherwise, negative.

Adjusting the threshold changes the balance between precision and recall:

• Lower threshold (e.g., 0.3): More cases predicted as positive → higher recall, lower precision
• Higher threshold (e.g., 0.7): Fewer cases predicted as positive → lower recall, higher precision

Calibration: Are predicted probabilities accurate? If the model says 30% probability for a class, do roughly 30% of those cases actually belong to that class? Miscalibrated models can mislead downstream decisions.

# Adjust threshold and see the effect on precision/recall
from sklearn.metrics import precision_recall_curve

y_pred_proba = clf.predict_proba(X_test)[:, 1]

# Compute precision and recall for different thresholds
precisions, recalls, thresholds = precision_recall_curve(y_test, y_pred_proba)

# Plot precision-recall curve
plt.figure(figsize=(10, 5))
plt.plot(recalls, precisions, linewidth=2, label='Precision-recall curve')
plt.xlabel("Recall (caught positives)")
plt.ylabel("Precision (positive predictions that are correct)")
plt.title("Precision-Recall Curve: trade-off when adjusting decision threshold")
plt.legend()
plt.grid(True, alpha=0.3)
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.tight_layout()
plt.show()

print("Key insight: higher recall requires lower precision, and vice versa.")
print("Choose threshold based on your use case's cost of false positives vs false negatives.")

Boundary complexity and overfitting

A simple linear boundary generalizes better than a complex, wiggly boundary — unless the true boundary is actually nonlinear. The goal is to match model complexity to problem complexity:

• Underfitting: Model boundary is too simple; poor training and test accuracy.
• Good fit: Model boundary is about right; good training and good test accuracy.
• Overfitting: Model boundary is too complex; excellent training accuracy, poor test accuracy.

Use validation curves to find the sweet spot:

from sklearn.model_selection import validation_curve

param_range = range(1, 20)
train_scores, val_scores = validation_curve(
    DecisionTreeClassifier(random_state=42),
    X, y,
    param_name="max_depth",
    param_range=param_range,
    cv=5,
    scoring="accuracy"
)

train_mean = train_scores.mean(axis=1)
val_mean = val_scores.mean(axis=1)

plt.figure(figsize=(10, 5))
plt.plot(param_range, train_mean, 'o-', label='Training score', linewidth=2)
plt.plot(param_range, val_mean, 's-', label='Cross-validation score', linewidth=2)
plt.xlabel("Max tree depth (model complexity)")
plt.ylabel("Accuracy")
plt.title("Validation Curve: find the complexity sweet spot")
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

# Identify best depth
best_depth = param_range[val_mean.argmax()]
print(f"Best max_depth: {best_depth} (highest cross-validation score)")

Probability calibration and decision thresholds in practice

Many classifiers output probabilities (e.g., logistic regression outputs P(class=1) ∈ [0, 1]). The default decision threshold is 0.5 — if P ≥ 0.5, predict class 1; else class 0. But this is not always optimal.

Calibration question: If the model predicts P = 0.3, does that mean ~30% of those examples are actually class 1?

# Check calibration using the calibration_curve utility
from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt

# Get predicted probabilities on a large test set
y_pred_proba = clf.predict_proba(X_test)[:, 1]

# Compute calibration curve
prob_true, prob_pred = calibration_curve(y_test, y_pred_proba, n_bins=10)

# Plot
plt.figure(figsize=(10, 5))
plt.plot([0, 1], [0, 1], 'k--', label='Perfectly calibrated')
plt.plot(prob_pred, prob_true, 'o-', label='Logistic regression')
plt.xlabel("Mean predicted probability")
plt.ylabel("Fraction of positives (actual)")
plt.title("Calibration Curve: are predicted probabilities accurate?")
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

# Interpretation:
# If the curve is above the diagonal, the model is overconfident (predicts 0.6, but <60% are actually positive)
# If below, the model is underconfident (predicts 0.6, but >60% are actually positive)

Adjusting threshold for precision/recall trade-off:

from sklearn.metrics import precision_recall_curve

# Find the threshold that maximizes F1 score
precisions, recalls, thresholds = precision_recall_curve(y_test, y_pred_proba)
f1_scores = 2 * (precisions * recalls) / (precisions + recalls + 1e-10)
best_threshold = thresholds[f1_scores.argmax()]

print(f"Best threshold for F1: {best_threshold:.3f}")
print(f"  Precision: {precisions[f1_scores.argmax()]:.3f}")
print(f"  Recall: {recalls[f1_scores.argmax()]:.3f}")

# For a custom cost (e.g., FP cost = $10, FN cost = $100):
fp_cost, fn_cost = 10, 100
cost_array = (1 - thresholds) * fp_cost + thresholds * fn_cost
best_cost_threshold = thresholds[cost_array.argmin()]
print(f"Best threshold for cost: {best_cost_threshold:.3f}")

Real-world case: fraud detection trade-offs

Credit card fraud dataset (illustrative estimates; real fraud rates are ~0.1%, fraud impact is thousands per incident):

| Threshold | Fraud Detection Rate | False Alarm Rate | Est. Net Savings | |-----------|--------|---------|---------| | 0.1 (catch all fraud) | 95% | 15% (20M false alarms/year) | $500M (fraud caught) – $200M (false alarm cost) = $300M | | 0.5 (default) | 75% | 2% (2.7M false alarms) | $375M – $27M = $348M | | 0.9 (strict) | 40% | 0.1% (135K false alarms) | $200M – $1.35M = $198M |

Interpretation: A threshold of 0.5 balances fraud prevention and customer friction. Lowering it to 0.1 catches more fraud but creates 7x more false alarms (and customer frustration). Raising it to 0.9 minimizes false alarms but lets half the fraud through. The choice depends on the business cost of each error type.

Common mistake: picking the wrong decision boundary for your problem

A complex boundary that fits training data perfectly is useless if it does not generalize. Always evaluate on a held-out test set, and always plot your decision boundaries (in 2D) or examine residuals (in higher dimensions) to sanity-check the model's logic.

If the boundary has unexplained wiggles or corners, it is probably overfitting. Simplify (reduce depth, increase regularization) and retest.

Another trap: assuming the default threshold (0.5) is optimal. In production, adjust the threshold to match your precision/recall trade-off, which depends on false positive and false negative costs. A 0.7 threshold might be perfect for one use case, terrible for another.

Advanced: AUC-ROC and imbalanced classification

Accuracy is misleading with imbalanced data. If 98% of examples are class 0, a model that always predicts 0 achieves 98% accuracy but catches no positive cases.

Use AUC-ROC (Area Under the Receiver Operating Characteristic curve) instead. It plots True Positive Rate (TPR = Recall) vs False Positive Rate (FPR = FP / N) at every threshold:

from sklearn.metrics import roc_auc_score, roc_curve
import matplotlib.pyplot as plt

# Get predicted probabilities
y_pred_proba = clf.predict_proba(X_test)[:, 1]

# Compute AUC
auc = roc_auc_score(y_test, y_pred_proba)
fpr, tpr, thresholds = roc_curve(y_test, y_pred_proba)

# Plot ROC curve
plt.figure(figsize=(10, 5))
plt.plot(fpr, tpr, label=f'ROC curve (AUC = {auc:.3f})')
plt.plot([0, 1], [0, 1], 'k--', label='Random classifier (AUC = 0.5)')
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC Curve")
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

print(f"AUC-ROC: {auc:.3f}")
print(f"  AUC = 0.5: Random classifier")
print(f"  AUC = 0.7: Fair classifier")
print(f"  AUC = 0.9: Excellent classifier")

AUC interpretation: A random classifier has AUC = 0.5 (the diagonal). A perfect classifier has AUC = 1.0. AUC = 0.7 is "fair," 0.8 is "good," 0.9+ is "excellent."

AUC is threshold-agnostic: it measures overall discrimination ability across all thresholds, unlike accuracy which is tied to a specific threshold.