Handling Imbalanced Datasets

The Accuracy Trap

You've trained a classifier on a dataset where 99% of the samples are class 0 (negative) and 1% are class 1 (positive). Your model predicts class 0 for every single sample. What accuracy do you get? 99%. Problem solved?

No. Your model is useless. It's failing the minority class entirely, which is often the one you actually care about—fraud cases, rare diseases, faulty parts, churn-at-risk customers. A naive accuracy metric gives you a false sense of success and can hide systematic failure on the class that matters most.

This is the fundamental challenge of imbalanced classification: class imbalance breaks the basic assumption that accuracy reflects how well your model works. When one class dominates, a trivial baseline achieves high accuracy by predicting the majority class alone. This becomes especially dangerous in production, where a high-accuracy model might be generating actual harm by failing to detect the rare but important cases your business depends on.

In this lesson, you'll learn why imbalance is dangerous, what metrics to use instead, and five concrete techniques to handle it: class weighting, oversampling (including SMOTE), undersampling, threshold tuning, and cost-sensitive ensemble methods. You'll see production-grade code for each approach, understand the computational and statistical tradeoffs, and learn how to choose the right strategy for your specific business constraints.

Why Standard Metrics Fail on Imbalanced Data

Let's say you're building a fraud detection model. Out of 10,000 transactions, 50 are fraudulent (0.5% fraud rate). You train a logistic regression model that predicts "not fraud" for everything. Your metrics:

• Accuracy: 99.5%
• Precision (on fraud class): undefined (no positive predictions)
• Recall (on fraud class): 0%
• F1-score: 0%

Accuracy is high but recall—the percentage of actual fraud cases you catch—is zero. Your business stakeholder will fire you for missing all the fraud. You've built a model that passes the test but fails in production.

The core issue: accuracy weights all errors equally and all classes equally. When class 1 is only 0.5% of the data, even detecting zero cases of class 1 gets you 99.5% accuracy. You need metrics that care about the minority class and reflect your business cost function.

Use these instead:

• Precision: Of the cases you flagged as positive, how many were actually positive? Matters when false positives are costly (e.g., alerting a customer of fraud they didn't commit, wasting investigator time, poor user experience).
• Recall: Of all actual positive cases, how many did you catch? Matters when false negatives are costly (e.g., missing real fraud, allowing a disease to progress untreated, shipping a faulty part).
• F1-score: Harmonic mean of precision and recall. A single score that penalizes you for trading one off against the other. Equally weights precision and recall.
• F-beta score: Weighted harmonic mean; use F2 (weight recall 2x higher) if false negatives are much more costly than false positives, or F0.5 (weight precision 2x) if false positives dominate.
• Precision-Recall curve (PR-AUC): A curve showing the tradeoff between precision and recall across all decision thresholds. More informative than ROC-AUC on imbalanced data because ROC-AUC can be artificially high even when recall on the minority class is poor.
• Matthews Correlation Coefficient (MCC): A balanced metric even for imbalanced data; ranges from -1 to +1, where +1 is perfect prediction.
• Confusion matrix by group: Always report true positives, false positives, true negatives, false negatives. Don't hide behind a single number. Transparency is your friend.

These are the honest metrics that expose when you're really just predicting the majority class. Using them requires discipline—your stakeholder may push back on a "lower" F1 score compared to "99% accuracy," but you must educate them on why the distinction matters.

Technique 1: Class Weighting

The simplest approach is to tell your classifier that errors on the minority class are more expensive. Most classifiers—logistic regression, random forests, neural networks—support a class_weight parameter that lets you assign higher penalty to minority-class errors. This is a form of cost-sensitive learning: you explicitly encode the cost of different error types into the loss function.

How it works: During training, when the model makes a mistake on a minority-class example, the loss contribution is multiplied by the class weight. A weight of 1.0 means normal cost; a weight of 10.0 means that error is 10x more expensive. The optimizer tries harder to avoid mistakes on highly-weighted classes.

Here's a concrete example with scikit-learn:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_recall_curve, auc, f1_score, confusion_matrix
from sklearn.datasets import make_classification

# Create synthetic imbalanced data: 98% class 0, 2% class 1
# Simulates a fraud detection scenario: 2% of transactions are fraudulent
X, y = make_classification(
    n_samples=10000,
    n_features=20,
    n_informative=10,
    n_redundant=5,
    weights=[0.98, 0.02],
    random_state=42
)

# Split into train/test with stratification to maintain imbalance ratio
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

print(f"Training set: {(y_train == 0).sum()} negatives, {(y_train == 1).sum()} positives")
print(f"Test set: {(y_test == 0).sum()} negatives, {(y_test == 1).sum()} positives")

# Model WITHOUT class weights (baseline)
model_unweighted = LogisticRegression(random_state=42, max_iter=200)
model_unweighted.fit(X_train, y_train)

# Model WITH class weights balanced to inverse class frequency
# 'balanced' automatically computes: weight_c = n_samples / (n_classes * n_samples_c)
# For 98/2 split: weight_0 ≈ 0.51, weight_1 ≈ 25.5 (minority class is 50x more expensive)
model_weighted = LogisticRegression(
    class_weight='balanced',
    random_state=42,
    max_iter=200
)
model_weighted.fit(X_train, y_train)

# Compare metrics on the minority class
from sklearn.metrics import recall_score, precision_score

y_pred_unweighted = model_unweighted.predict(X_test)
y_pred_weighted = model_weighted.predict(X_test)

print("\n=== Unweighted Model ===")
print(f"Recall (catch fraud): {recall_score(y_test, y_pred_unweighted):.3f}")
print(f"Precision (false alarms): {precision_score(y_test, y_pred_unweighted):.3f}")
print(f"F1-score: {f1_score(y_test, y_pred_unweighted):.3f}")

print("\n=== Weighted Model ===")
print(f"Recall (catch fraud): {recall_score(y_test, y_pred_weighted):.3f}")
print(f"Precision (false alarms): {precision_score(y_test, y_pred_weighted):.3f}")
print(f"F1-score: {f1_score(y_test, y_pred_weighted):.3f}")

# Show confusion matrices
print("\nUnweighted confusion matrix:")
print(confusion_matrix(y_test, y_pred_unweighted))

print("\nWeighted confusion matrix:")
print(confusion_matrix(y_test, y_pred_weighted))

The class_weight='balanced' parameter automatically computes weights inversely proportional to class frequencies. The minority class errors become much more expensive, forcing the model to work harder at identifying class 1.

You can also specify custom weights. If you know from your business that missing a fraud case (false negative) costs you $100, but incorrectly flagging a legitimate transaction costs $5 (investigating a false alarm), you can set class_weight={0: 5, 1: 100} to reflect that asymmetry directly.

Trade-offs:

• ✓ Simple to implement (one parameter)
• ✓ No data manipulation required
• ✓ Works with any algorithm supporting class weights
• ✗ May sacrifice recall on majority class (but that's intentional)
• ✗ If imbalance is extreme (< 0.1%), weighting alone may not suffice; combine with other techniques

When to use: Use class weighting as your first approach on any imbalanced problem. It's computationally cheap and usually effective. If you need better minority-class performance, combine with resampling (next techniques).

Technique 2: Oversampling and SMOTE

Another approach: duplicate or synthetically generate samples from the minority class until the dataset is more balanced. The obvious way—just copying minority samples—risks overfitting because you're duplicating exact samples, and the model can memorize them. A better way is SMOTE (Synthetic Minority Over-sampling Technique), which generates synthetic examples by interpolating between existing minority-class samples in feature space.

How SMOTE works: For each minority-class sample, find its k nearest neighbors (also minority class). Then generate synthetic samples at random points along the line segments between that sample and its neighbors. This creates new examples that are plausible variations of the minority class, not duplicates.

from imblearn.over_sampling import SMOTE
from sklearn.metrics import roc_auc_score
import matplotlib.pyplot as plt

# SMOTE generates synthetic samples to balance the dataset
smote = SMOTE(random_state=42, k_neighbors=5, sampling_strategy='minority')
X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)

# After SMOTE, we have roughly equal class counts
print(f"Original class distribution: {np.bincount(y_train)}")
print(f"After SMOTE: {np.bincount(y_train_resampled)}")
print(f"Original size: {X_train.shape[0]}, After SMOTE: {X_train_resampled.shape[0]}")

# Train a model on the resampled data
model_smote = LogisticRegression(random_state=42, max_iter=200)
model_smote.fit(X_train_resampled, y_train_resampled)

# Evaluate on ORIGINAL test set (not resampled!)
y_pred_smote = model_smote.predict(X_test)
y_proba_smote = model_smote.predict_proba(X_test)[:, 1]

print("\n=== SMOTE Model ===")
print(f"Recall (minority class): {recall_score(y_test, y_pred_smote):.3f}")
print(f"Precision (minority class): {precision_score(y_test, y_pred_smote):.3f}")
print(f"F1-score: {f1_score(y_test, y_pred_smote):.3f}")
print(f"ROC-AUC: {roc_auc_score(y_test, y_proba_smote):.3f}")

# CRITICAL: Never SMOTE your test set!
# This is a common mistake that makes metrics look good but misleads you
# in production

Critical point: Apply SMOTE only to the training set. Your test set must reflect the real class distribution you'll encounter in production. If you SMOTE your test set, your metrics will be artificially inflated and fail in production.

Variants and tuning:

• k_neighbors: How many neighbors to use for interpolation. Default is 5. Lower (3-5) for sparse data, higher (5-10) for dense regions.
• sampling_strategy: Oversample to what ratio? Default 'minority' brings minority up to match majority (1:1). You can also use a fraction (0.5 oversample to 50% minority, leaving class imbalance).
• ADASYN (Adaptive Synthetic Sampling): Similar to SMOTE but focuses on harder-to-learn boundary examples. Can be better for complex decision boundaries.

Trade-offs:

• ✓ Avoids overfitting compared to simple duplication
• ✓ Creates diverse synthetic examples
• ✓ Computationally lighter than some alternatives
• ✗ Synthetic data may not capture real minority-class diversity
• ✗ Fails if minority class has very few samples (< 10)
• ✗ Can create unrealistic synthetic samples in high-dimensional spaces
• ✗ Doesn't help if the real issue is concept drift (see lesson 23)

When to use: Use SMOTE when your minority class is moderately-sized (10-1000 samples) and you want to explicitly rebalance the training set. For very small minorities (< 10 samples), stick with class weighting. For very large minorities, undersampling may be faster.

Technique 3: Undersampling

The inverse of oversampling: remove samples from the majority class to reduce the imbalance. This is faster than generating synthetic data, but you lose information about the majority class.

from imblearn.under_sampling import RandomUnderSampler

# Randomly remove majority-class samples to achieve a 1:1 ratio
undersampler = RandomUnderSampler(random_state=42)
X_train_undersampled, y_train_undersampled = undersampler.fit_resample(X_train, y_train)

print(f"After undersampling: {np.bincount(y_train_undersampled)}")

# Train on undersampled data
model_undersample = LogisticRegression(random_state=42, max_iter=200)
model_undersample.fit(X_train_undersampled, y_train_undersampled)

y_pred_undersample = model_undersample.predict(X_test)
print("Undersampling recall (minority class):", recall_score(y_test, y_pred_undersample))

Trade-off: You throw away majority-class data, which may cost you decision boundary information if your majority class is heterogeneous. Undersampling works best when you have abundant majority-class data and can afford to discard some.

When to use: Use undersampling when your dataset is very large and the majority class is over-represented (e.g., 1 million majority vs. 10,000 minority samples). Reduces computational cost and can still work well.

Technique 4: Threshold Tuning and Cost-Aware Optimization

All classifiers output a probability or score, not a hard class label. Most use a default threshold of 0.5: if P(class=1) >= 0.5, predict 1; else predict 0. On imbalanced data, this threshold is arbitrary and often wrong. Tuning the threshold lets you directly trade off precision and recall to match your actual business cost function.

The key insight: optimal threshold depends on business costs, not on the data. A model trained on fraud data doesn't inherently "know" whether false positives or false negatives are more expensive in your business.

from sklearn.metrics import precision_recall_curve, roc_auc_score, f1_score

# Get probability predictions on the test set
y_prob = model_weighted.predict_proba(X_test)[:, 1]  # Probability of class 1

# Compute precision-recall curve
precisions, recalls, thresholds = precision_recall_curve(y_test, y_prob)

# Option 1: Find threshold that maximizes F1 (equal weight precision/recall)
f1_scores = 2 * (precisions[:-1] * recalls[:-1]) / (precisions[:-1] + recalls[:-1] + 1e-10)
best_f1_idx = np.argmax(f1_scores)
best_threshold_f1 = thresholds[best_f1_idx]

print(f"Threshold for max F1: {best_threshold_f1:.3f}")
print(f"F1 at this threshold: {f1_scores[best_f1_idx]:.3f}")

# Option 2: Find threshold based on business cost function
# Example: false negative (missing fraud) costs $100, false positive costs $5
cost_fn = 100  # Cost of false negative
cost_fp = 5    # Cost of false positive

# For each threshold, calculate total cost
total_costs = []
for idx, threshold in enumerate(thresholds):
    y_pred_threshold = (y_prob >= threshold).astype(int)
    tn = ((y_test == 0) & (y_pred_threshold == 0)).sum()
    fp = ((y_test == 0) & (y_pred_threshold == 1)).sum()
    fn = ((y_test == 1) & (y_pred_threshold == 0)).sum()
    tp = ((y_test == 1) & (y_pred_threshold == 1)).sum()

    total_cost = cost_fp * fp + cost_fn * fn
    total_costs.append(total_cost)

best_cost_idx = np.argmin(total_costs)
best_threshold_cost = thresholds[best_cost_idx]

print(f"\nThreshold for min business cost: {best_threshold_cost:.3f}")
print(f"Expected cost at this threshold: ${total_costs[best_cost_idx]:.0f}")

# Compare different thresholds
print("\n=== Threshold Comparison ===")
for threshold in [0.2, 0.3, 0.5, 0.7, 0.9]:
    y_pred = (y_prob >= threshold).astype(int)
    p = precision_score(y_test, y_pred) if (y_pred == 1).sum() > 0 else 0
    r = recall_score(y_test, y_pred)
    fp = ((y_test == 0) & (y_pred == 1)).sum()
    fn = ((y_test == 1) & (y_pred == 0)).sum()
    business_cost = cost_fp * fp + cost_fn * fn

    print(f"Threshold {threshold:.1f}: Precision {p:.3f}, Recall {r:.3f}, Cost ${business_cost:.0f}")

Example output (illustrative estimate):

• Threshold 0.2: Precision 0.45, Recall 0.95, Cost $2500
• Threshold 0.3: Precision 0.60, Recall 0.92, Cost $2100
• Threshold 0.5: Precision 0.75, Recall 0.78, Cost $2800
• Threshold 0.7: Precision 0.85, Recall 0.55, Cost $5000
• Threshold 0.9: Precision 0.92, Recall 0.20, Cost $9000

In this example (cost_fn=$100, cost_fp=$5), the optimal threshold is around 0.3, not the default 0.5. This threshold minimizes expected business loss.

When to use threshold tuning:

• After training any probability-based classifier
• When you can quantify business costs (even roughly)
• For high-stakes decisions where the cost asymmetry is large
• Combined with SMOTE, class weighting, and undersampling for maximum effectiveness

Trade-offs:

• ✓ No retraining required
• ✓ Can change in production based on new cost information
• ✓ Directly optimizes for business metrics
• ✗ Requires you to estimate business costs (not always easy)
• ✗ Doesn't help if the model's recall is fundamentally too low (use other techniques first)

Technique Comparison and Selection Matrix

Here's a detailed comparison of all techniques, helping you choose the right one(s) for your situation:

| Technique | Imbalance Ratio | Dataset Size | Speed | Minority Recall | Accuracy Loss | Best For | |-----------|-----------------|--------------|-------|-----------------|---------------|----------| | Class weighting | Any | Any | ⭐⭐⭐ Fast | Medium | Low | First-pass baseline; any algorithm | | SMOTE | 1-10% | 1k-100k | ⭐⭐ Medium | High | Low-Medium | Moderate imbalance; enough minority samples | | Undersampling | < 1% | > 100k | ⭐⭐⭐ Fast | High | Medium | Large datasets; majority abundant | | Threshold tuning | Any | Any | ⭐⭐⭐ Fast | High | None (recall-precision tradeoff) | Post-hoc optimization; you have biz costs | | Combined (SMOTE + Class Weight) | 1-5% | 10k-50k | ⭐⭐ Medium | Very High | Low | Maximum minority recall |

Selection logic:

1. Always start with class weighting—it's free and works with any algorithm.
2. If minority recall < 80% and ratio < 5%, add SMOTE to the training pipeline.
3. If dataset > 500k and imbalance is extreme, use undersampling to cut computational cost.
4. Always tune thresholds on a validation set using precision-recall curves, not the default 0.5.
5. Monitor precision AND recall on test data, never just accuracy.

In practice, the best approach often combines 2-3 techniques: weighted training + SMOTE resampling + threshold tuning.

Real-World Case Study: Credit Card Fraud Detection

Let's walk through a realistic scenario combining all techniques. You're building a fraud detection model for a credit card company.

Problem setup:

• 50 million transactions per year
• Fraud rate: 0.1% (50,000 fraudulent transactions)
• Cost of missed fraud: customer dispute, chargeback, replacement card = ~$200 per case
• Cost of false alarm: customer frustration, call center overhead = ~$2 per case
• Your goal: minimize total annual cost

Year 1: Initial model You collect 6 months of data (25M transactions, ~25k fraudulent). You build a logistic regression model with class_weight='balanced' and test on recent 2 weeks of data (100k transactions, ~100 fraudulent).

Results:

• Default threshold (0.5): 78% recall, 99% precision → $4M annual fraud cost + $200k false alarm cost = $4.2M
• Optimized threshold (0.15): 92% recall, 72% precision → $1.6M annual fraud cost + $5.6M false alarm cost = $7.2M ❌ (worse!)
• Optimized threshold (0.30): 88% recall, 85% precision → $2.4M annual fraud cost + $2.6M false alarm cost = $5M (better, but still high)

The model struggles because 0.1% imbalance is extreme. You need stronger techniques.

Year 2: Enhanced pipeline You add SMOTE to the training set and manually tune the decision threshold using your cost function:

# Add SMOTE to training
from imblearn.pipeline import Pipeline as ImbPipeline
from imblearn.over_sampling import SMOTE

pipeline = ImbPipeline([
    ('smote', SMOTE(sampling_strategy=0.2, random_state=42)),  # Oversample to 20% minority
    ('logistic', LogisticRegression(class_weight='balanced', max_iter=200))
])

pipeline.fit(X_train, y_train)
y_prob = pipeline.predict_proba(X_test)[:, 1]

# Find threshold that minimizes cost
cost_fn = 200  # fraud cost
cost_fp = 2    # false alarm cost

best_threshold = find_best_threshold(y_test, y_prob, cost_fn, cost_fp)

Results:

• Optimized threshold (0.08): 94% recall, 65% precision → $1.2M annual fraud cost + $11M false alarm cost = $12.2M (over-flagging!)
• Optimized threshold (0.20): 89% recall, 80% precision → $2.2M annual fraud cost + $4M false alarm cost = $6.2M
• Optimized threshold (0.25): 86% recall, 85% precision → $2.8M annual fraud cost + $2.8M false alarm cost = $5.6M ✓ (better)

SMOTE + class weighting improved recall but you're still leaving money on the table. You need even stronger techniques.

Year 3: Ensemble + threshold optimization You train a gradient boosting model (better capacity) with class weights, use SHAP for explanation, and implement an "escalation policy":

• Green zone (confidence > 0.9): Auto-approve 99% of transactions
• Yellow zone (confidence 0.30-0.90): Run through fraud rules engine (additional signals)
• Red zone (confidence < 0.30): Flag for manual review or decline

This hybrid approach balances automation (speed) with precision (fewer false alarms).

Final results:

• Model recall (on manual review set): 97%
• False alarm rate on auto-approved: 0.05%
• Annual cost: $1.5M fraud + $1.5M false alarms + $0.5M manual review labor = $3.5M ✓✓ (30% reduction from baseline)

Lessons from this case study:

1. Start simple (class weighting) and measure
2. Understand your cost function; don't optimize accuracy blindly
3. Combine multiple techniques—no single technique is a silver bullet
4. Consider hybrid human+model systems for high-stakes decisions
5. Measure business cost, not just classification metrics

Combining Multiple Techniques for Real-World Data

In practice, you rarely use just one technique. A realistic imbalanced-data pipeline combines multiple approaches:

from imblearn.pipeline import Pipeline as ImbPipeline
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from imblearn.over_sampling import SMOTE
from imblearn.under_sampling import RandomUnderSampler

# Chain: undersample majority, oversample minority, then train
pipeline = ImbPipeline([
    ('undersample', RandomUnderSampler(random_state=42, sampling_strategy=0.5)),
    ('oversample', SMOTE(random_state=42, k_neighbors=5)),
    ('scaler', StandardScaler()),
    ('model', LogisticRegression(class_weight='balanced', random_state=42, max_iter=200))
])

# Fit on training data
pipeline.fit(X_train, y_train)

# Evaluate
y_pred = pipeline.predict(X_test)
print("F1 score:", f1_score(y_test, y_pred))

# The pipeline automatically applies resampling only to training folds during cross-validation,
# never touching the test set. This is the correct way to do it.

This combined approach gives you flexibility: undersample the majority to reduce dataset size (faster training), oversample the minority to fill in gaps (better minority coverage), then apply class weights as a regularizer (cost-sensitive learning).

When Class Imbalance Is Actually a Feature, Not a Bug

In some domains, class imbalance is not a problem to solve but a reality to respect. Example: fraud detection. Fraud really is ~0.1% of transactions in the wild. If you oversample fraud in training to get a 50/50 split, your trained model will predict fraud way too often in production because the base rates don't match.

In such cases:

• Don't rebalance your training set. Keep the original distribution.
• Use metrics that respect the imbalance: precision (of cases you flag as fraud, how many are real?), recall (of real fraud, how many do you catch?), or PR-AUC.
• Tune your decision threshold based on your business cost, not on class balance.
• Monitor precision-recall curves, not accuracy.

This approach works because your test set reflects production, and your metrics (precision, recall) are well-defined for imbalanced data.

Real-World Performance Impact Summary

Here's a summary table showing realistic performance improvements from the techniques discussed:

| Scenario | Baseline (None) | Class Weight | SMOTE | Undersampling | Threshold Tuning | Combined | |----------|-----------------|-------------|-------|----------------|------------------|----------| | Fraud detection (0.1% fraud) | Recall 20%, Precision 95% | 70% recall, 45% precision | 85% recall, 30% precision | 80% recall, 35% precision | 78% recall, 50% precision | 90% recall, 40% precision | | Loan default (5% default) | Recall 40%, Precision 80% | 65% recall, 70% precision | 75% recall, 60% precision | 70% recall, 65% precision | 68% recall, 75% precision | 80% recall, 65% precision | | Disease diagnosis (2% disease) | Recall 15%, Precision 85% | 55% recall, 65% precision | 70% recall, 50% precision | 65% recall, 55% precision | 60% recall, 70% precision | 78% recall, 58% precision |

These are illustrative estimates; actual improvements depend on your data and model architecture. The key insight: combining techniques beats any single approach, but at the cost of added complexity.

Implementation Effort vs. Benefit

| Technique | Implementation Time | Maintenance Burden | Benefit/Effort Ratio | Recommended For | |-----------|-------------------|------------------|---------------------|-----------------| | Class weighting | < 1 hour | Minimal | ⭐⭐⭐⭐⭐ Excellent | Always (first baseline) | | SMOTE | 2-4 hours | Low (tune k_neighbors) | ⭐⭐⭐⭐ Great | Moderate imbalance | | Undersampling | 1-2 hours | Minimal | ⭐⭐⭐ Good | Very large datasets | | Threshold tuning | 2-3 hours | Low (adjust based on costs) | ⭐⭐⭐⭐ Great | Cost-based optimization | | Combined pipeline | 8-16 hours | Medium (cross-validation) | ⭐⭐⭐⭐⭐ Excellent | Production systems |

Common Mistake: Resampling Before Train-Test Split

Many practitioners resample their entire dataset before splitting into train and test. This causes data leakage: information from test samples influences the synthetic samples created, and your test set is no longer representative of real data.

Always resample after the split:

# WRONG: Resampling before split
X_resampled, y_resampled = smote.fit_resample(X, y)
X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled)

# RIGHT: Resampling after split, on training data only
X_train, X_test, y_train, y_test = train_test_split(X, y)
X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)

Only the training set should be resampled. Your test set must always reflect the true class distribution you'll encounter in production.

Also avoid: resampling and then using the resampled data for hyperparameter tuning (cross-validation). Resample inside each fold, not before, so test folds stay balanced to the true distribution. Scikit-learn's imblearn.pipeline.Pipeline handles this automatically.