Feature Engineering Before the Model Ever Sees Data

Feature Engineering: The Often-Overlooked Advantage

Raw data is almost never ready for a model. The difference between a mediocre model and an excellent one often comes down not to the algorithm, but to how well you've engineered your input features. Feature engineering is the art and science of transforming raw observations into numerical representations that make the learning problem easier for your model to solve.

Feature engineering accounts for roughly 70% of the effort in real ML projects, yet is often glossed over in favor of chasing complex algorithms. A simple linear model on well-engineered features often outperforms a sophisticated ensemble on raw features—a principle captured in the old adage: "garbage in, garbage out."

The pipeline above shows the typical flow: raw data enters from the left, passes through cleaning, encoding, scaling, feature construction, and finally feature selection—all fit on training data exclusively. This separation is not optional; it is critical to avoid contaminating your test results with information learned from the test set itself.

Understanding the Cost of Data Leakage

Before you write a single line of preprocessing code, internalize one principle: fit your preprocessor on training data only. If you compute the mean of a column or learn how to encode a categorical variable using the entire dataset (training + test), you have leaked information from the test set into your preprocessing. Your model then has an unfair advantage, and your evaluation metrics become unreliable.

The correct workflow is:

1. Fit preprocessing steps on the training set.
2. Transform the training set using the learned parameters.
3. Transform the test set using the same learned parameters (without re-fitting).

Many practitioners make the mistake of normalizing the entire dataset before splitting it into train and test. This inflates performance metrics by 2–5% in typical scenarios (illustrative estimate based on common benchmarks), making the model appear better than it actually is in production.

Why Data Leakage Is Subtle

Consider standardization (z-score normalization). The formula is:

x_scaled = (x - mean(X)) / std(X)

If you compute mean and std on the full dataset including test data, you've optimistically scaled the test set using statistics that include information from the test set itself. When the model is deployed on truly new data, that new data won't have been included in computing these statistics, and the preprocessor will behave differently. This difference causes a gap between your measured performance and true production performance—often catastrophic for imbalanced or time-series data.

Quantifying the Leakage Impact

Research on feature engineering pipelines shows that premature scaling before train/test split inflates R² scores by an illustrative estimate of 1–3 percentage points on standard benchmarks. For classification, precision and recall can shift by similar margins. In production, when the model sees data that wasn't used to compute the scaler's parameters, performance drops noticeably.

Handling Missing Values

Real-world datasets are rarely complete. The imputation strategy you choose affects not just model accuracy but your ability to interpret results. Data with 10–20% missing values are common in production, yet improper handling can silently degrade model performance.

Common strategies:

• Mean/median imputation: Replace missing numeric values with the column's mean or median. Simple and fast, but ignores relationships between features and may bias downstream statistics.
• Forward/backward fill: For time series, carry the last observed value forward or the next value backward. Effective for temporal data where values are correlated in time.
• Iterative imputation (KNN, MICE): Use neighboring rows or other features to predict missing values via regression. More sophisticated but computationally expensive and can introduce subtle biases if not careful.
• Deletion: Drop rows or columns with missing values. Only viable when missingness is very rare (<5%), otherwise you discard useful training data.
• Indicator variable approach: Create a binary "was_missing" flag for each feature with missingness, then impute with a sentinel value (e.g., -1 or 0). Allows the model to learn whether missingness itself is predictive.

Here is a scikit-learn implementation:

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestClassifier

# Create synthetic data with missing values
X_train = pd.DataFrame({
    'age': [25, 30, np.nan, 45, 50],
    'income': [40000, np.nan, 60000, 80000, 100000],
    'city': ['NYC', 'LA', 'NYC', 'Chicago', 'LA']
})
y_train = [0, 1, 0, 1, 1]

X_test = pd.DataFrame({
    'age': [np.nan, 35, 52],
    'income': [75000, 90000, 120000],
    'city': ['NYC', 'Boston', 'LA']
})
y_test = [1, 0, 1]

# Define preprocessing for numeric and categorical columns separately
numeric_features = ['age', 'income']
numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),  # Fit on training data
    ('scaler', StandardScaler())
])

categorical_features = ['city']
categorical_transformer = Pipeline(steps=[
    ('encoder', OneHotEncoder(handle_unknown='ignore'))
])

# Combine both transformers
preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)
    ])

# Build full pipeline
full_pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', RandomForestClassifier(n_estimators=100, random_state=42))
])

# Fit on training data only
full_pipeline.fit(X_train, y_train)

# Transform and predict on test data
predictions = full_pipeline.predict(X_test)
print("Predictions:", predictions)

Notice that the SimpleImputer and StandardScaler are fit only when you call full_pipeline.fit(X_train, y_train). When you call predict(X_test), they apply the learned imputation mean and scaling parameters to the test set without refitting.

Comparison of Imputation Strategies:

| Strategy | Missingness Tolerance | Computation Cost | Preserves Relationships | Bias | Best For | |----------|-----|------|------|------|----------| | Mean/Median | < 20% | Very fast | No | Underestimates variance | Quick baseline | | KNN imputation | < 30% | Moderate | Partial (local) | Low if k well-chosen | Mixed numeric/cat | | MICE (iterative) | < 40% | High | Yes (global) | Low (multivariate) | MCAR data, strong dependencies | | Deletion | < 5% | Instant | N/A | High (loss of signal) | Sparse missingness only | | Indicator + imputation | < 50% | Fast | No, but preserves missingness signal | Depends on sentinel value | When missingness is predictive |

Encoding Categorical Variables

Categorical features (city names, product types, customer segments) are text or labels that models cannot directly use. You must convert them to numbers. The choice of encoding has profound effects on model interpretability and performance.

Common approaches:

• One-Hot Encoding: Create a binary column for each unique category. Works well when the number of categories is small (< 50). For n categories, you get n-1 or n binary columns (depending on whether you drop one to avoid collinearity).
• Ordinal Encoding: Assign integers (0, 1, 2, ...) to categories. Use only if the categories have a natural order (e.g., education level: "high school" < "bachelor" < "master"). Risky if applied to unordered categories because the model may learn spurious ordinal relationships.
• Target Encoding: Replace each category with the mean target value for that category in the training set. Powerful for high-cardinality features but risks severe overfitting. Must be fit on training data only and ideally regularized via smoothing.
• Frequency Encoding: Replace each category with its frequency (count or proportion in the training set). Simple and space-efficient but loses categorical information. Useful for very high-cardinality features (1000+ unique values).

Target Encoding in Depth

Target encoding (also called mean encoding) replaces each categorical value with the target mean of that category:

encoded_value = mean(y | category = c)

For example, if city "NYC" has a target mean of 0.65 (65% of NYC records are positive), all NYC records are encoded as 0.65.

Why it works: The model no longer needs to learn latent associations; the feature directly encodes target correlation.

The risk: With rare categories (few samples per category), the mean is unreliable. A category with only 1 sample will have a mean of exactly 0 or 1, which likely doesn't generalize. Smoothing (regularization) mitigates this:

encoded_value = (n_samples_in_category * mean(y | category) + global_mean * smoothing_strength) / (n_samples_in_category + smoothing_strength)

Higher smoothing pushes rare categories closer to the global mean. Standard smoothing strength is 1–10.

Here is a comparison of encoding methods:

| Method | Cardinality | Interpretability | Overfitting Risk | Curse of Dimensionality | |--------|-----|------|------|------| | One-Hot | Low (< 50) | High (easy to debug) | Low | Yes, explodes with many categories | | Ordinal | Low & ordered | High (but risky if misapplied) | Low | No | | Target (unsmoothed) | High (50–10K) | Medium (coefficients = target means) | Very high (rare categories) | No | | Target (smoothed) | High (50–10K) | Medium | Low (regularized) | No | | Frequency | Very high (10K+) | Medium | Low | No |

For our age/income/city example above, the OneHotEncoder creates three columns (one for each city) and sets them to 0 or 1. The handle_unknown='ignore' parameter ensures that if a city appears in the test set that did not appear in training, it is encoded as all zeros, preventing errors.

Scaling and Normalization

Many algorithms (linear regression, logistic regression, SVMs, neural networks) are sensitive to the magnitude of input features. A feature ranging from 0 to 1000000 dominates one ranging from 0 to 1 in gradient-based optimization. Without scaling, gradient descent updates in high-magnitude features drown out updates in low-magnitude features, causing slow or biased convergence.

Common scaling techniques:

• Standardization (Z-score normalization): Subtract the mean and divide by the standard deviation. Results in features with mean 0 and standard deviation 1.

  x_scaled = (x - mean(X_train)) / std(X_train)
  

  This is the default and works well for most algorithms. The mean and std are fit on training data and applied to test/deployment data.

• Min-Max Scaling: Scale to a fixed range, typically [0, 1].

  x_scaled = (x - min(X_train)) / (max(X_train) - min(X_train))
  

  Useful when you need bounded outputs or when features have known meaningful ranges. Risk: a single outlier in the test set can break this scaling if the test min/max exceed the training bounds.

• Robust Scaling: Like standardization, but uses median and interquartile range (IQR) instead of mean and std.

  x_scaled = (x - median(X_train)) / IQR(X_train)
  

  Better for data with outliers because median and IQR are less influenced by extreme values. If your data has known outliers, use this instead of StandardScaler.

The StandardScaler fits on the training set's mean and standard deviation, then applies the same transformation to the test set. This ensures the test set is on the same scale as the training set, even if it has a slightly different mean or variance.

from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

# Simulated feature with wide range and outliers
X_train = np.array([[1], [2], [3], [4], [5], [100]])  # Outlier at 100
X_test = np.array([[1.5], [2.5], [102]])  # Test set includes unseen extremes

# Standard scaling
scaler_std = StandardScaler()
X_train_std = scaler_std.fit_transform(X_train)
X_test_std = scaler_std.transform(X_test)  # Uses training mean/std

# Robust scaling
scaler_robust = RobustScaler()
X_train_robust = scaler_robust.fit_transform(X_train)
X_test_robust = scaler_robust.transform(X_test)

# Min-Max scaling (risky with outliers)
scaler_minmax = MinMaxScaler()
X_train_minmax = scaler_minmax.fit_transform(X_train)
X_test_minmax = scaler_minmax.transform(X_test)

print("Standard-scaled test:", X_test_std)
print("Robust-scaled test:", X_test_robust)
print("MinMax-scaled test:", X_test_minmax)  # Will have values outside [0, 1]!

Comparison of scaling methods:

| Method | Preserves Outliers | Bounded Output | When to Use | |--------|------|----|----| | StandardScaler | Yes (pulls up std) | No | Default; most algorithms | | MinMaxScaler | Yes (affects range) | Yes [0,1] | When bounded output needed, clean data | | RobustScaler | No (uses IQR) | No | Data with outliers, financial data |

Feature Construction and Interaction Terms

Sometimes the original features are insufficient. Creating new features by combining existing ones can dramatically improve model performance.

Techniques:

• Polynomial features: If x is age, create x^2 (age squared) to capture non-linear relationships.
• Interaction terms: Multiply two features, e.g., age * income to capture how these factors interact.
• Binning: Convert continuous values into categorical ranges (e.g., age groups: 18–25, 26–35, etc.).
• Domain-specific features: Extract day of week from a timestamp, country from an IP address, etc.

Example:

from sklearn.preprocessing import PolynomialFeatures

# Create polynomial and interaction features
poly_transformer = PolynomialFeatures(degree=2, include_bias=False)
# Fit on training data
X_train_poly = poly_transformer.fit_transform(X_train[numeric_features])
# Transform test data using learned polynomial basis
X_test_poly = poly_transformer.transform(X_test[numeric_features])

print("Original shape:", X_train[numeric_features].shape)
print("Polynomial features shape:", X_train_poly.shape)

For our 2-feature example (age and income), degree 2 creates: age, income, age^2, income^2, and age * income—five features total. This added expressiveness comes at the cost of more parameters to fit and more risk of overfitting if not regularized (see the lesson on regularization).

Feature Selection: Reducing Noise and Dimensionality

Not all features are equally useful. Some may be noisy, redundant, or irrelevant. Removing them reduces overfitting, speeds up training, and improves interpretability.

Methods:

• Univariate selection: Compute the statistical relationship (e.g., correlation, mutual information) between each feature and the target, independently. Fast but ignores feature interactions.
• Recursive Feature Elimination (RFE): Train a model, remove the least important feature, retrain, repeat. More computationally expensive but captures feature dependencies.
• L1 regularization (Lasso): The model itself drives weak features toward zero coefficients. Covered in the regularization lesson.

Here is a practical example using SelectKBest:

from sklearn.feature_selection import SelectKBest, f_classif

# Select top 3 features based on univariate statistical test
selector = SelectKBest(score_func=f_classif, k=3)
# Fit on training data
X_train_selected = selector.fit_transform(X_train_poly, y_train)
# Transform test data
X_test_selected = selector.transform(X_test_poly)

# Get the names of selected features
selected_indices = selector.get_support(indices=True)
print("Selected feature indices:", selected_indices)

The f_classif function computes an F-statistic for each feature, measuring how much its value differs between classes. Features with high F-scores are selected.

A Complete, Production-Ready Pipeline

Bringing it all together:

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression

numeric_features = ['age', 'income']
categorical_features = ['city']

# Numeric pipeline: impute, then scale
numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

# Categorical pipeline: one-hot encode
categorical_transformer = Pipeline(steps=[
    ('encoder', OneHotEncoder(handle_unknown='ignore', sparse_output=False))
])

# Combine transformers
preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)
    ])

# Full pipeline with feature selection and model
full_pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('feature_selection', SelectKBest(f_classif, k=5)),
    ('classifier', LogisticRegression(max_iter=1000, random_state=42))
])

# Train
full_pipeline.fit(X_train, y_train)

# Evaluate
train_score = full_pipeline.score(X_train, y_train)
test_score = full_pipeline.score(X_test, y_test)
print(f"Train accuracy: {train_score:.3f}, Test accuracy: {test_score:.3f}")

This pipeline ensures that imputation, scaling, encoding, and feature selection are all fit on the training data and applied identically to the test data. No leakage occurs.

Working with Imbalanced Classes

In many real-world datasets, classes are imbalanced—one class is much rarer than others. For example, fraud detection datasets might be 99% legitimate transactions and 1% fraud. A naive model that predicts "legitimate" for everything achieves 99% accuracy but is useless.

Feature engineering approaches for imbalance:

1. Stratified sampling: When splitting into train/test, preserve class proportions in each split.

   from sklearn.model_selection import train_test_split
   X_train, X_test, y_train, y_test = train_test_split(
       X, y, test_size=0.2, stratify=y, random_state=42
   )
   

2. Synthetic Minority Oversampling (SMOTE): Generate synthetic samples for the minority class. However, fit SMOTE only on training data, then apply to test set:

   from imblearn.over_sampling import SMOTE
   smote = SMOTE(random_state=42)
   X_train_balanced, y_train_balanced = smote.fit_resample(X_train, y_train)
   # Note: Do not apply SMOTE to test set (test set should be left as-is)
   

3. Class weights: Let your classifier weight minority samples more heavily. Most scikit-learn classifiers support class_weight='balanced':

   clf = LogisticRegression(class_weight='balanced')
   

The distinction is crucial: oversampling (like SMOTE) artificially inflates training set size, which can distort evaluation metrics if accidentally applied to the test set.

Feature Interactions and Domain Knowledge

Raw features often lack expressiveness. Smart feature engineering leverages domain knowledge to create features that capture important relationships.

Example: Real Estate Pricing

# Raw features
price = df['price']
bedrooms = df['bedrooms']
square_feet = df['square_feet']

# Engineered features
df['price_per_sqft'] = df['price'] / df['square_feet']
df['rooms_per_sqft'] = df['bedrooms'] / (df['square_feet'] / 1000)
df['has_yard'] = (df['lot_size'] > 5000).astype(int)

These derived features often capture the true underlying relationship better than raw values alone. In a housing price model, "price per square foot" is far more predictive than raw price because it normalizes for house size.

Example: Time Series Features

import pandas as pd

df['date'] = pd.to_datetime(df['date'])
df['day_of_week'] = df['date'].dt.dayofweek
df['month'] = df['date'].dt.month
df['is_weekend'] = df['day_of_week'].isin([5, 6]).astype(int)
df['days_since_start'] = (df['date'] - df['date'].min()).dt.days

For time series, extracting temporal patterns (day of week, seasonality, trend) often yields more predictive features than raw timestamps.

Quantifying Feature Importance

After engineering features, which ones actually matter? Feature importance scores help you understand which features drive predictions and which are noise.

from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance

# Train a model
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# Built-in feature importance (mean decrease in impurity)
importances = model.feature_importances_
feature_names = X_train.columns
sorted_idx = np.argsort(importances)[::-1]

print("Feature importances (top 10):")
for i in range(min(10, len(feature_names))):
    idx = sorted_idx[i]
    print(f"{feature_names[idx]}: {importances[idx]:.4f}")

# Permutation importance (more robust, tests impact on held-out data)
perm_importance = permutation_importance(model, X_test, y_test, n_repeats=10, random_state=42)
print("\nPermutation importances:")
for i, importance in enumerate(perm_importance.importances_mean):
    print(f"{feature_names[i]}: {importance:.4f}")

Permutation importance is often more reliable: it shuffles each feature and measures the resulting drop in model performance. Large drops indicate important features; small drops suggest the feature is redundant or noisy.

Case Study: Feature Engineering on a Real Churn Prediction Dataset

To illustrate how feature engineering impacts real-world performance, consider a telecommunications churn prediction task. Raw data includes:

• Numeric: monthly charges ($0–$150), tenure (0–72 months), data usage (0–500 GB)
• Categorical: contract type ("Month-to-month", "One year", "Two year"), internet service ("Fiber optic", "DSL", "None")
• Missing: 5–8% of service-related features are sometimes NULL

Without feature engineering (baseline): One-hot encode contract and service types, delete missing values, feed to logistic regression → validation AUC 0.68.

With intelligent feature engineering:

1. Imputation: Use KNN imputation (k=5) for service features, preserving relationships.
2. Feature construction: Create charge_per_month_active = monthly_charges / max(tenure, 1), capturing customer intensity. Create high_usage_flag = (data_usage > 100 GB).
3. Target encoding: Encode contract and service types using smoothed target encoding (smoothing_strength=5) since categories are unordered but predictive.
4. Scaling: StandardScaler on all numeric features.
5. Feature selection: SelectKBest with f_classif to keep top 12 features.

Result: Validation AUC improves from 0.68 to 0.76 (illustrative estimate)—an 11% relative improvement in AUC without changing the algorithm. The gain comes entirely from better feature representation.

from sklearn.impute import KNNImputer
from sklearn.preprocessing import StandardScaler, FunctionTransformer
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
import pandas as pd
import numpy as np

# Simulate a churn dataset
np.random.seed(42)
n = 500
X_raw = pd.DataFrame({
    'monthly_charges': np.random.uniform(18, 118, n),
    'tenure_months': np.random.uniform(0, 72, n),
    'data_usage_gb': np.random.uniform(0, 500, n),
    'contract_type': np.random.choice(['Month-to-month', 'One year', 'Two year'], n),
    'internet_service': np.random.choice(['Fiber optic', 'DSL', 'None'], n),
    'extra_feature_1': np.random.randn(n),  # Sometimes missing
})
# Introduce missing values
X_raw.loc[np.random.choice(n, 30, replace=False), 'extra_feature_1'] = np.nan

y = (X_raw['monthly_charges'] > 80).astype(int) + np.random.binomial(1, 0.2, n)
y = (y > 0).astype(int)

# Split
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_raw, y, test_size=0.2, random_state=42)

# Define feature engineering pipeline
numeric_features = ['monthly_charges', 'tenure_months', 'data_usage_gb', 'extra_feature_1']
categorical_features = ['contract_type', 'internet_service']

# Custom feature construction function
def construct_features(X):
    X = X.copy()
    X['charge_per_month'] = X['monthly_charges'] / (X['tenure_months'] + 1)
    X['high_usage'] = (X['data_usage_gb'] > 100).astype(int)
    return X[numeric_features + ['charge_per_month', 'high_usage']]

# Numeric pipeline: impute (KNN), construct features, scale
numeric_transformer = Pipeline(steps=[
    ('imputer', KNNImputer(n_neighbors=5)),
    ('feature_builder', FunctionTransformer(construct_features)),
    ('scaler', StandardScaler())
])

# Categorical: one-hot encode (or use target encoding for production)
categorical_transformer = Pipeline(steps=[
    ('encoder', OneHotEncoder(handle_unknown='ignore', sparse_output=False))
])

# Combine
preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)
    ])

# Full pipeline with feature selection and model
full_pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('feature_selection', SelectKBest(f_classif, k=10)),
    ('model', LogisticRegression(max_iter=1000, random_state=42))
])

full_pipeline.fit(X_train, y_train)
train_score = full_pipeline.score(X_train, y_train)
test_score = full_pipeline.score(X_test, y_test)
print(f"Train accuracy: {train_score:.3f}, Test accuracy: {test_score:.3f}")

This example demonstrates the power of systematic feature engineering: imputation preserves signal, feature construction adds domain knowledge, encoding handles categoricals properly, and selection removes noise.

Common mistake

The most frequent pitfall is computing statistics (mean, std, category counts) on the full dataset before splitting:

# WRONG: This leaks test information
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_all_scaled = scaler.fit_transform(X_all)  # X_all includes test data!
X_train_scaled, X_test_scaled = train_test_split(X_all_scaled, ...)

The scaler learned the global mean and std, which includes the test set. When your model is deployed on truly new data, it will not have this unfair advantage, and performance will drop.

The fix: Always fit on training data, then transform:

# RIGHT: No leakage
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Use ColumnTransformer and Pipeline to automate this and eliminate the risk of forgetting.