Ensemble Methods: Bagging, Boosting, and Stacking

Three ensemble strategies: diversity is strength

An ensemble combines multiple models to produce a stronger prediction than any single model alone. The key insight: if models make different mistakes on different examples, averaging them reduces variance without sacrificing bias.

Variance reduction via diversity: If model A makes error +0.5 on example i and model B makes error -0.5, averaging gives error 0. But if both make error +0.5, averaging gives +0.5 — no improvement. Ensemble methods succeed when errors are uncorrelated.

Three core strategies exist:

Bagging (Bootstrap Aggregating)

Concept: Train M models in parallel on M random subsets (bootstrap samples) of the training data, drawn with replacement. Prediction is the average (or majority vote) of all M models.

Formal: For m = 1 to M:

1. Sample n examples with replacement from training data → S_m
2. Train model h_m on S_m
3. Final prediction: ŷ = (1/M) · Σₘ h_m(x)

Why it works: Each bootstrap sample is different, so each model learns a slightly different boundary. Errors are uncorrelated, so averaging reduces variance. The key: variance reduction = (1/M) · original_variance if errors are fully independent.

Best for: Reducing variance of high-capacity, unstable models (e.g., deep decision trees).

Example: Random Forest. Each tree sees a bootstrap sample of rows and features, trained independently. All trees vote on the final prediction.

Boosting

Concept: Train M models sequentially. Each new model h_m focuses on correcting the errors of the ensemble so far. Combine them as a weighted sum: ŷ = Σₘ α_m · h_m(x), where α_m reflects model m's strength.

Formal (Gradient Boosting):

1. Initialize h₀(x) = mean(y)
2. For m = 1 to M:
   • Compute residuals r_m = y - h_(x)
   • Train h_m to predict r_m
   • Update: h_m(x) ← h_(x) + learning_rate · h_m(x)

Why it works: By focusing learning on hard examples (those with large residuals), boosting reduces bias first, then variance. Each model specializes in what the ensemble got wrong.

Best for: Reducing bias (underfitting) and improving accuracy on difficult cases.

Example: Gradient Boosting. Each shallow tree predicts the residuals of the previous ensemble. The final model is the sum of all trees' predictions (scaled by learning rate).

Stacking

Concept: Train K diverse base models (using different algorithms or feature subsets) on training data, then train a meta-model (level-1 learner) on the predictions of those base models.

Formal:

1. Split training data into parts A and B
2. Train base models h₁, ..., h_K on part A
3. Generate meta-features: ŷ_ = [h₁(x_i), ..., h_K(x_i)] for examples in part B
4. Train meta-model g on (ŷ_, y_B)
5. Final prediction: ŷ = g([h₁(x), ..., h_K(x)])

Why it works: The meta-model learns which base models to trust for which types of inputs. This is especially powerful when base models have different strengths — e.g., one tree is great on linear patterns, another neural network captures nonlinearity.

Best for: Combining fundamentally different model types (e.g., trees + SVM + linear models) when you have enough data to train a meta-learner.

Bagging: Random Forest example

Random Forest trains many decision trees on random subsets of data, then averages their predictions.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, mean_squared_error
import matplotlib.pyplot as plt

# Generate classification data
np.random.seed(42)
X, y = make_classification(n_samples=1000, n_features=20, n_informative=10,
                           n_redundant=5, n_classes=2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Single deep decision tree (baseline: high variance, prone to overfitting)
single_tree = DecisionTreeClassifier(max_depth=10, random_state=42)
single_tree.fit(X_train, y_train)
single_pred = single_tree.predict(X_test)
single_acc = accuracy_score(y_test, single_pred)

# Random Forest: ensemble of many trees
forest = RandomForestClassifier(n_estimators=100, max_depth=10, random_state=42)
forest.fit(X_train, y_train)
forest_pred = forest.predict(X_test)
forest_acc = accuracy_score(y_test, forest_pred)

print("=" * 70)
print("BAGGING: SINGLE TREE vs RANDOM FOREST")
print("=" * 70)
print(f"\nSingle decision tree (max_depth=10):")
print(f"  Test accuracy: {single_acc:.3f}")
print()
print(f"Random Forest (100 trees, max_depth=10):")
print(f"  Test accuracy: {forest_acc:.3f}")
print()
print(f"Improvement: {(forest_acc - single_acc)*100:+.1f} percentage points")
print(f"  → Bagging reduced variance by averaging 100 diverse trees.")
print()

# Feature importance: average importance across all trees
importances = forest.feature_importances_
top_features = np.argsort(importances)[-5:][::-1]
print(f"Top 5 important features:")
for i, feat_idx in enumerate(top_features, 1):
    print(f"  {i}. Feature {feat_idx}: {importances[feat_idx]:.4f}")

Key properties of bagging:

• Reduces variance by decorrelating errors across models
• Each tree sees a different bootstrap sample, so they overfit in different ways
• Out-of-bag (OOB) estimation: Each training sample is excluded from ~37% of bootstrap samples, allowing built-in validation without a separate test set
• Naturally parallelizable: all trees train independently

Boosting: Gradient Boosting example

Gradient Boosting trains trees sequentially, each correcting the previous trees' residuals.

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import classification_report

# Single tree for comparison
single_tree_gb = DecisionTreeClassifier(max_depth=3, random_state=42)
single_tree_gb.fit(X_train, y_train)
single_pred_gb = single_tree_gb.predict(X_test)
single_acc_gb = accuracy_score(y_test, single_pred_gb)

# Gradient Boosting: sequential ensemble
# learning_rate (shrinkage) controls how much each tree contributes (lower = more conservative)
gb_model = GradientBoostingClassifier(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=3,
    random_state=42
)
gb_model.fit(X_train, y_train)
gb_pred = gb_model.predict(X_test)
gb_acc = accuracy_score(y_test, gb_pred)

# Probability predictions (for additional analysis)
gb_pred_proba = gb_model.predict_proba(X_test)[:, 1]

print("=" * 70)
print("BOOSTING: SINGLE TREE vs GRADIENT BOOSTING")
print("=" * 70)
print(f"\nSingle decision tree (max_depth=3):")
print(f"  Test accuracy: {single_acc_gb:.3f}")
print()
print(f"Gradient Boosting (100 trees, max_depth=3, learning_rate=0.1):")
print(f"  Test accuracy: {gb_acc:.3f}")
print()
print(f"Improvement: {(gb_acc - single_acc_gb)*100:+.1f} percentage points")
print(f"  → Boosting added weak learners sequentially, each correcting prior errors.")
print()
print(f"Classification Report (Gradient Boosting on test set):")
print(classification_report(y_test, gb_pred))

How gradient boosting works (simplified):

1. Train tree 1 on the full dataset → predictions h₁(x), residuals r₁ = y - h₁(x)
2. Train tree 2 on residuals r₁ → corrections c₂(x)
3. Update ensemble: H(x) = h₁(x) + learning_rate · c₂(x)
4. Repeat: fit tree 3 on new residuals, add its corrections, etc.

Hyperparameters to tune:

• n_estimators: Number of trees (more = slower training, usually better accuracy up to a point)
• learning_rate (shrinkage): Step size for adding corrections (lower = more conservative, less overfitting)
• max_depth: Depth of each tree (shallow trees are weak learners, easy to combine)

Bagging vs Boosting: strengths, trade-offs, and bias-variance decomposition

| Property | Bagging | Boosting | |----------|---------|----------| | Training | Parallel (all trees independent) | Sequential (each tree depends on prior) | | What it reduces | Variance (overfitting of high-capacity models) | Both bias and variance (but bias reduction is faster) | | Base learner | High-capacity, unstable (deep trees) | Weak learners (shallow trees, depth 1-3) | | Overfitting risk | Low; individual trees can be deep (max_depth=15+) | Medium; sensitive to learning_rate and n_estimators | | Speed | Fast training (parallelizable) | Slower (sequential, not easily parallelized) | | Inference speed | Moderate (must run all M trees) | Moderate (must run all M trees) | | Feature importance | Clear (average importance across trees) | Less clear (importance compounds across trees) | | Robustness to outliers | Medium (outliers affect many trees similarly) | Low (boosting upweights outliers) | | Typical accuracy gain vs single tree | +5–15 percentage points (illustrative estimate) | +10–25 percentage points (illustrative estimate) | | When to use | Fast training, want interpretability, linear/simple patterns | Maximum accuracy, complex patterns, willing to tune hyperparameters |

Bias-variance decomposition: Expected test error = Bias² + Variance + Irreducible noise

• Bagging: Reduces Variance (high-capacity models have high variance); Bias stays same
• Boosting: Reduces Bias first (weak learners have high bias); Variance reduction comes later via ensemble

This explains why boosting often outperforms bagging on underfitting problems (high bias), but bagging is safer for overfitting-prone models.

Real-world case study: Kaggle competition results

Dataset: Titanic survival prediction (1,309 passengers, 11 features after preprocessing)

Task: Predict which passengers survived based on age, fare, cabin class, sex, etc.

Results (illustrative estimates based on benchmark leaderboards; actual competition results vary):

| Model | Accuracy | Precision (survive) | Recall (survive) | Training Time | Production Friendly | |-------|----------|--------|---------|---------|---------| | Logistic regression (baseline) | 0.77 | 0.71 | 0.58 | <1 s | Very good | | Single decision tree | 0.81 | 0.76 | 0.67 | <1 s | Good | | Random Forest (100 trees) | 0.84 | 0.80 | 0.72 | 2 s | Fair (many trees) | | Gradient Boosting (100 trees) | 0.87 | 0.82 | 0.79 | 5 s | Fair (slower) | | Stacking (RF + GB + LR meta-learner) | 0.88 | 0.84 | 0.81 | 15 s | Poor (complexity) |

Interpretation:

• Bagging (Random Forest) improved accuracy by +3 points vs. single tree, with good parallelization.
• Boosting (Gradient Boosting) improved accuracy by +6 points vs. single tree, with better recall (caught more survivors).
• Stacking added marginal +1 point for 3x the complexity — not worth it for this problem.

In production, the trade-off is often between Random Forest (fast, interpretable) and Gradient Boosting (slower, more accurate). Stacking is justified only if the +1% accuracy gain has clear business value.

Stacking: a meta-model on top

Stacking trains a meta-model (or meta-learner) on the predictions of base models. The meta-model learns which base models to trust for each input.

Example: stacking with diverse base models

from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import StackingClassifier

# Define base models (diverse types: tree, SVM, linear)
base_models = [
    ('rf', RandomForestClassifier(n_estimators=50, max_depth=5, random_state=42)),
    ('svc', SVC(kernel='rbf', probability=True, random_state=42)),
    ('lr', LogisticRegression(max_iter=1000, random_state=42))
]

# Define meta-model: a logistic regression that combines base predictions
meta_model = LogisticRegression(max_iter=1000, random_state=42)

# Create stacking classifier
stacking_clf = StackingClassifier(
    estimators=base_models,
    final_estimator=meta_model,
    cv=5  # 5-fold cross-validation to generate meta-features
)

stacking_clf.fit(X_train, y_train)
stacking_pred = stacking_clf.predict(X_test)
stacking_acc = accuracy_score(y_test, stacking_pred)

print("=" * 70)
print("STACKING: META-MODEL ON BASE MODELS")
print("=" * 70)
print(f"\nBase model accuracies:")
for name, model in base_models:
    base_pred = model.fit(X_train, y_train).predict(X_test)
    print(f"  {name:15s}: {accuracy_score(y_test, base_pred):.3f}")
print()
print(f"Stacking classifier (meta-model combines base predictions):")
print(f"  Test accuracy: {stacking_acc:.3f}")
print()
print(f"Why stacking helps:")
print(f"  Each base model has strengths/weaknesses on different examples.")
print(f"  The meta-model learns which base models to trust for each input,")
print(f"  effectively combining their strengths.")

Key insight: Stacking works best when base models are diverse — trained on different algorithms, features, or even different subsets of data. If all base models make the same mistakes, stacking cannot fix them.

Voting: simple ensemble without meta-learning

A simpler alternative to stacking is voting: train diverse base models and let them vote. VotingClassifier in scikit-learn supports hard voting (majority class) and soft voting (average probabilities).

from sklearn.ensemble import VotingClassifier

# Define base models
voting_clf = VotingClassifier(
    estimators=[
        ('rf', RandomForestClassifier(n_estimators=50, random_state=42)),
        ('svc', SVC(kernel='rbf', probability=True, random_state=42)),
        ('gb', GradientBoostingClassifier(n_estimators=50, random_state=42))
    ],
    voting='soft'  # Use probability voting instead of majority vote
)

voting_clf.fit(X_train, y_train)
voting_acc = accuracy_score(y_test, voting_clf.predict(X_test))

print(f"Voting Classifier (soft voting on 3 base models):")
print(f"  Test accuracy: {voting_acc:.3f}")

When ensemble methods succeed and fail

Ensembles work well when:

• Base models are diverse (different algorithms or data subsets) — if all models make the same mistakes, ensembling helps little
• You have enough data to train multiple models
• Interpretability is less critical (ensembles are black-box-ier than single models)

Ensembles struggle when:

• All base models are highly correlated (they overfit to the same patterns)
• You have very limited training data (spreading data across multiple models leaves each one underfitted)
• Latency is critical (inference requires running all base models, slowing predictions)

Practical guidance: which to choose?

Use Random Forest (bagging) when:

• You want fast training and prediction
• Interpretability via feature importance is valuable
• You have compute for parallelization

Use Gradient Boosting when:

• You need maximum accuracy and are willing to tune hyperparameters
• Training time is less critical
• You suspect underfitting (weak individual models)

Use Stacking when:

• You have diverse model types (trees, linear, neural networks)
• You have enough data to train a meta-model
• The extra complexity is justified by accuracy gains

Start with Random Forest for a fast, interpretable baseline. If accuracy is insufficient, try Gradient Boosting. Only resort to stacking if the first two do not deliver adequate performance.

Hyperparameter tuning for ensemble methods

Bagging / Random Forest hyperparameters:

• n_estimators: Number of trees. More = slower but usually better (up to a point). Typical: 100–1000.
• max_depth: Depth of each tree. Deeper = higher variance, shallower = higher bias. Typical: 10–20 for Random Forest (allow overfitting in individual trees).
• min_samples_split: Minimum samples to split a node. Higher = simpler trees, less overfitting. Typical: 2–10.
• max_features: Number of features to consider per split. Lower = more diverse trees. Typical: sqrt(p) or log(p).

Boosting / Gradient Boosting hyperparameters:

• n_estimators: Number of trees. More = better, but slower and overfitting risk. Typical: 50–500.
• learning_rate (shrinkage): Scales each tree's contribution. Lower (e.g., 0.01) = slower but usually better generalization. Typical: 0.001–0.1.
• max_depth: Depth of weak learners. Shallow trees (depth 2–5) work best for boosting. Deeper trees ≈ stronger individual learners, but boosting is less effective.
• subsample: Fraction of samples per tree. Lower = more stochasticity, less overfitting. Typical: 0.5–1.0.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier

# Hyperparameter grid for Random Forest
rf_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [10, 15, 20],
    'min_samples_split': [2, 5, 10]
}

rf_search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    rf_grid,
    cv=5,  # 5-fold cross-validation
    scoring='accuracy',
    n_jobs=-1  # Parallel
)
rf_search.fit(X_train, y_train)
print(f"Best Random Forest params: {rf_search.best_params_}")
print(f"Best CV accuracy: {rf_search.best_score_:.3f}")

# Hyperparameter grid for Gradient Boosting
gb_grid = {
    'n_estimators': [50, 100, 200],
    'learning_rate': [0.001, 0.01, 0.1],
    'max_depth': [2, 3, 4]
}

gb_search = GridSearchCV(
    GradientBoostingClassifier(random_state=42),
    gb_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1
)
gb_search.fit(X_train, y_train)
print(f"Best Gradient Boosting params: {gb_search.best_params_}")
print(f"Best CV accuracy: {gb_search.best_score_:.3f}")

# Compare final test performance
print(f"\nTest set performance:")
print(f"  Random Forest: {rf_search.score(X_test, y_test):.3f}")
print(f"  Gradient Boosting: {gb_search.score(X_test, y_test):.3f}")

Common mistake: tuning too much and overfitting the ensemble

Ensemble hyperparameters (learning rate, number of estimators, base learner depth) can easily be tuned to perfection on the training set, leading to overfitting. Always use cross-validation or a held-out validation set to monitor out-of-sample performance, and stop tuning when validation performance plateaus.

Similarly, adding more trees does not always improve test accuracy — if n_estimators is already large, boosting may be overfitting. Monitor training and validation loss curves to catch this. Look for:

• Training loss keeps decreasing, validation plateaus or rises → classic overfitting
• Both decrease in tandem → good generalization

Also avoid hyperparameter overfitting: if you search over a huge grid, you will find hyperparameters that work great on the validation set but fail on truly held-out test data. Use a separate validation set (not the same set used for early stopping or curve plotting).

Key takeaway: ensemble as a debugging and exploration tool

Ensembles are not just for accuracy. They are also diagnostic tools:

• Feature importance from Random Forest: Which features matter most?
• Boosting residuals: On which examples does the ensemble struggle?
• Base model disagreement in stacking: Do base models agree on predictions? High disagreement suggests the problem is hard.

Use ensembles to understand your data and model behavior, not just to squeeze out 1% accuracy gains.