Explainability Methods: SHAP, LIME, and Beyond

The Black Box Problem

You've trained a neural network on loan application data. It predicts whether to approve a mortgage. A customer is denied, and they demand to know why. You have three choices:

1. Tell them "the neural network said so" (useless, legally risky)
2. Trace through all 50 layers and 2 million weights to manually reconstruct the decision (impossible)
3. Use an explainability method to approximate why the model made that decision

Most models—deep neural networks, gradient boosting, random forests—are black boxes. They work well, but you can't point to a single weight or rule and say "this is why." In high-stakes domains like lending, healthcare, and hiring, black boxes are unacceptable. Regulators demand explainability. Customers demand fairness. You need tools.

This lesson covers two major explainability methods: SHAP (SHapley Additive exPlanations), which ranks feature importance using game-theoretic Shapley values, and LIME (Local Interpretable Model-agnostic Explanations), which approximates a complex model locally with a simple linear model. You'll see real code for both, learn their strengths and limitations, and understand when to use each.

Global vs. Local Explanations

Before diving into SHAP and LIME, you need to distinguish two types of explanations:

Global explanations answer: "Across my entire dataset, which features are most important to my model's decisions?" Use this to understand your model's overall behavior, debug systematic biases, and validate that it's learning the right patterns.

Local explanations answer: "For this specific prediction, why did my model decide as it did?" Use this to explain individual decisions to users, understand edge cases, and catch errors on particular samples.

A model might learn that income is globally important for loans, but for a specific applicant, employment history might be the deciding factor. Global and local importance can differ.

Standard approaches like permutation feature importance or coefficients give you global importance. SHAP and LIME are designed for local explanations, though SHAP can also aggregate to global importance.

SHAP: Game-Theoretic Feature Attribution

SHAP values are a principled way to attribute a model's prediction to its input features using game theory. Imagine each feature is a player in a cooperative game trying to predict an outcome. The Shapley value tells you how much each player contributes to the final prediction, fairly accounting for interactions and coalition effects.

How SHAP works mathematically: For each feature, you compute its marginal contribution across all possible coalitions of other features. The Shapley value is the average of these marginal contributions. In plain language: "If this feature appeared, how much would it change the model's prediction (on average, given all possible other feature combinations)?"

Types of SHAP explainers:

• TreeExplainer: Fast (polynomial time) for tree-based models (XGBoost, LightGBM, Random Forest)
• KernelExplainer: Model-agnostic but slower; uses weighted local regression to approximate Shapley values
• DeepExplainer: For neural networks; uses DeepLIFT algorithm

Here's a production-grade example using different explainers:

import shap
import numpy as np
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import time

# Create and train models
X, y = make_classification(
    n_samples=5000,
    n_features=20,
    n_informative=10,
    n_redundant=5,
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Tree-based model (can use TreeExplainer for speed)
rf_model = RandomForestClassifier(n_estimators=50, random_state=42)
rf_model.fit(X_train, y_train)

# LinearModel (simpler baseline)
lr_model = LogisticRegression(random_state=42, max_iter=1000)
lr_model.fit(X_train, y_train)

# SHAP explanation: Tree Explainer (fast, tree-specific)
print("=== TreeExplainer (RandomForest) ===")
start = time.time()
tree_explainer = shap.TreeExplainer(rf_model)
shap_values_tree = tree_explainer.shap_values(X_test[:50])  # Explain 50 samples
elapsed_tree = time.time() - start
print(f"Time to explain 50 samples: {elapsed_tree:.3f}s")

# SHAP explanation: Kernel Explainer (model-agnostic, slower)
print("\n=== KernelExplainer (LogisticRegression) ===")
start = time.time()
# For KernelExplainer, use a small background sample for efficiency
background = X_train[:100]
kernel_explainer = shap.KernelExplainer(
    lr_model.predict_proba,
    data=shap.sample(background, 50)
)
shap_values_kernel = kernel_explainer.shap_values(X_test[:50])  # Explain 50 samples
elapsed_kernel = time.time() - start
print(f"Time to explain 50 samples: {elapsed_kernel:.3f}s (2-10x slower than TreeExplainer)")

# Interpret global feature importance
mean_abs_shap_tree = np.abs(shap_values_tree[1]).mean(axis=0) if isinstance(shap_values_tree, list) else np.abs(shap_values_tree).mean(axis=0)
feature_importance_order = np.argsort(mean_abs_shap_tree)[::-1]

print("\n=== Global Feature Importance (top 5) ===")
for idx in feature_importance_order[:5]:
    print(f"Feature {idx}: {mean_abs_shap_tree[idx]:.4f}")

# Interpret a single prediction (local explanation)
test_idx = 0
prediction = rf_model.predict_proba(X_test[test_idx:test_idx+1])[0]
shap_vals_single = shap_values_tree[1][test_idx] if isinstance(shap_values_tree, list) else shap_values_tree[test_idx]

print(f"\n=== Single Prediction Explanation ===")
print(f"Predicted probability (class 1): {prediction[1]:.3f}")
print(f"Base value (avg prediction): {tree_explainer.expected_value[1]:.3f}")

# Show top contributing features
top_features = np.argsort(np.abs(shap_vals_single))[::-1][:5]
print("Top 5 contributing features:")
for feature_idx in top_features:
    direction = "↑ increases" if shap_vals_single[feature_idx] > 0 else "↓ decreases"
    print(f"  Feature {feature_idx}: {shap_vals_single[feature_idx]:+.4f} {direction}")

Key SHAP concepts:

• Base value: The model's average prediction across training data. Represents the starting point before any features push it up or down.
• Positive SHAP value: Feature pushes prediction toward positive class
• Negative SHAP value: Feature pushes prediction toward negative class
• Magnitude: How much this feature influenced this prediction

Trade-offs:

• ✓ Theoretically principled (satisfies key axioms)
• ✓ Works on any model via KernelExplainer
• ✓ Captures feature interactions
• ✓ Produces both local and global explanations (average absolute values)
• ✗ Computationally expensive: KernelExplainer O(2^n) can take seconds per sample
• ✗ TreeExplainer is fast but only works on tree models
• ✗ Assumes feature independence (correlation in features can mislead)
• ✗ Values are model-relative, not truly causal

When to use:

• Batch analysis: Pre-compute explanations for a set of representative samples
• Post-hoc analysis and fairness audits (lesson 24) where latency is not critical
• When you need global feature importance for model validation
• NOT for real-time explanations (unless you cache results)

LIME: Local Linear Approximations

LIME takes a different approach: instead of using game theory, it approximates your complex model locally with a simple linear model that is inherently interpretable.

Here's the idea: around a specific prediction you're trying to explain, perturb the input slightly, get predictions from your black-box model on those perturbed inputs, then fit a simple linear model (logistic regression) to the relationship between perturbations and predictions. This linear model is your local approximation, and its coefficients tell you which features matter for that specific prediction.

import lime
import lime.lime_tabular
from sklearn.ensemble import RandomForestClassifier

# Train a model (same as before)
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# Create a LIME explainer for tabular data
explainer = lime.lime_tabular.LimeTabularExplainer(
    training_data=X_train,
    feature_names=[f"Feature {i}" for i in range(X_train.shape[1])],
    class_names=["Class 0", "Class 1"],
    mode='classification'
)

# Explain a single prediction
test_sample = X_test[0]
explanation = explainer.explain_instance(
    data_row=test_sample,
    predict_fn=model.predict_proba,
    num_features=5  # Show top 5 features
)

# Print explanation
explanation.show_in_notebook()  # In Jupyter
# Or: explanation.as_list()  # Get as list of tuples

# Extract the local linear model's coefficients
local_prediction_coeffs = explanation.as_list()
for feature_name, coeff in local_prediction_coeffs:
    print(f"{feature_name}: {coeff:.4f}")

LIME's output might be: "Feature 2 > 0.5 (+0.42), Feature 7 <= 2.0 (+0.18), Feature 5 > 10 (-0.25)". These are the decision rules in the local linear model that approximate your black box's behavior in that region.

Strengths:

• Intuitive: the explanation is a simple linear model you can reason about
• Fast: you only evaluate your black box a few hundred times per explanation (vs. SHAP's many evaluations)
• Handles high-dimensional data well
• Can work with any model, including neural networks and image classifiers

Limitations:

• The local linear approximation is only valid near the sample being explained; the "neighborhood" is arbitrary
• Less principled than SHAP; no strong theoretical guarantees
• Relies on perturbation strategy (how do you perturb features?), which can affect results
• Can give misleading explanations if the local approximation is poor
• Doesn't naturally capture feature interactions as well as SHAP

When to use: Use LIME when you need explanations quickly in production, or when speed is more important than theoretical rigor. LIME is practical and usually sufficient for understanding individual predictions.

SHAP vs. LIME vs. Permutation Importance: Detailed Comparison

| Aspect | SHAP (TreeExplainer) | SHAP (KernelExplainer) | LIME | Permutation Importance | |--------|-----|-----|------|-----| | Speed per sample | ⭐⭐⭐ Fast (ms) | ⭐ Slow (1-10s) | ⭐⭐ Medium (100-500ms) | ⭐⭐⭐ Fast (batch) | | Theory | Game-theoretic (Shapley values) | Game-theoretic approx. | Heuristic local linear | Empirical importance | | Local explanations | Yes, per-sample | Yes, per-sample | Yes, per-sample | No | | Global explanations | Yes (average |SHAP|) | Yes (average |SHAP|) | No direct | Yes (variable importance) | | Captures interactions | ✓ Yes | ✓ Yes | Weak | ✓ Yes | | Works on any model | Trees only | ✓ Yes (slow) | ✓ Yes | ✓ Yes | | Interpretability of output | Marginal contribution | Marginal contribution | Local linear model | % decrease in accuracy | | When to use | Trees/XGBoost, batch analysis | Complex models, fairness audits | Production explanations | Baseline importance |

Decision matrix:

1. Real-time production explanation (< 200ms): Use LIME or cached SHAP
2. Batch analysis for fairness audits: Use SHAP (any explainer)
3. Feature importance for model validation: Use TreeExplainer (if tree-based) or permutation importance
4. Explain individual edge cases: Use LIME (quick) or SHAP (principled)
5. High-stakes decisions requiring audit trail: Use SHAP with full documentation

Cost in practice (illustrative estimate) on a 10-feature model with 1M monthly predictions:

• LIME: 100 explanations/month (edge cases only) → ~1 server-minute/month
• SHAP (Kernel): Same 100 explanations → ~100 server-minutes/month (100x more)
• TreeExplainer: 100k explanations/month (frequent) → ~5 server-minutes/month

Critical Caveat: Explanations Are Approximations, Not Truth

Both SHAP and LIME produce explanations by approximating or locally modeling your black box. These approximations are useful for understanding and debugging, but they are not the ground truth about how your model works internally.

Here's why this matters: A malicious actor could potentially craft adversarial examples that have misleading explanations—the explanation looks reasonable, but the model's true decision boundary is different. Or, a model could be learning spurious correlations that happen to align with your explanations.

Use explanations to:

• Build intuition about your model
• Debug unexpected predictions
• Validate that your model is learning sensible patterns
• Communicate to non-technical stakeholders why a decision was made

Do not use explanations to:

• Assume you fully understand the model's learned decision boundary
• Trust that the model is causal (correlation != causation)
• Ignore the need for other forms of validation (fairness audits, adversarial testing, human review)

In high-stakes domains (healthcare, lending, criminal justice), always pair explainability methods with:

• Baseline comparisons (does your model beat a simple rule?)
• Fairness audits across demographic groups (see lesson 24)
• Human review of the most consequential decisions
• An appeals/override process

Validating Explanations Against Domain Expertise

The most important step: validate that your explanations match domain expertise. Here's how:

# Validation: compare explanation to expert judgment
import pandas as pd

def validate_explanations(model, X_test, explanations, domain_expert_rules):
    """
    Validate if explanations align with known domain rules.

    domain_expert_rules: list of ("feature_name", "direction") tuples
    e.g., [("age", "positive"), ("income", "positive"), ("debt", "negative")]
    """
    mismatches = []

    for i, explanation in enumerate(explanations):
        # Extract top features from SHAP/LIME explanation
        top_features = explanation[:3]  # Top 3 features

        for feature_name, expected_direction in domain_expert_rules:
            # Find this feature in the explanation
            for exp_feature, imp in top_features:
                if feature_name in exp_feature:
                    actual_direction = "positive" if imp > 0 else "negative"

                    if actual_direction != expected_direction:
                        mismatches.append({
                            'sample_idx': i,
                            'feature': feature_name,
                            'expected': expected_direction,
                            'actual': actual_direction,
                            'importance': imp
                        })

    if mismatches:
        df_mismatch = pd.DataFrame(mismatches)
        print(f"⚠️  Found {len(mismatches)} explanation mismatches:")
        print(df_mismatch.head(10))
        return False  # Explanations don't match domain expertise
    else:
        print("✓ Explanations align with domain expertise")
        return True

# Example domain rules for a loan model
domain_rules = [
    ("income", "positive"),       # Higher income → more likely to approve
    ("debt_to_income", "negative"), # Higher debt → less likely to approve
    ("credit_score", "positive"),   # Higher score → more likely to approve
    ("age", "positive"),            # Older age (more stable) → more likely to approve
]

validation_passed = validate_explanations(model, X_test, explanations, domain_rules)

Key principle: If 30%+ of explanations contradict domain expertise, your explanation method is unreliable. Debug and retrain.

Advanced Use Cases: Feature Interaction and Time-Series Explanations

Both SHAP and LIME have extensions for more complex scenarios.

Feature interactions: SHAP can measure not just individual feature importance, but also interactions—how much does feature A's contribution depend on feature B's value? This is crucial for understanding nonlinear models.

# Interaction SHAP values (more expensive to compute)
# Shows which features interact
shap_interaction_values = explainer.shap_interaction_values(X_test[:10])

# Plot a summary of interactions
shap.summary_plot(shap_interaction_values, X_test[:10], plot_type="dot")

Time-series explanations: For sequences (e.g., predicting next stock price from price history), standard SHAP/LIME struggle because they assume feature independence. Research is ongoing in this area; for now, simpler approaches work: attention weights (if you have a transformer), gradient-based saliency, or LIME on aggregated features.

Case Study: Detecting Misleading Explanations

Scenario: You've deployed a LIME explainer for a loan approval model. A rejected applicant sees: "Income <= 30k increased denial probability by +0.45." They assume the model is discriminatory. Is it?

The trap: The explanation looks reasonable, but may not reflect the model's true decision boundary. Here's how to validate:

# Step 1: Collect suspicious explanations
# (explanations where the direction seems wrong or unexpected)

# Step 2: Compare explanation to model behavior
applicant = {...loan_data...}
true_prediction = model.predict(applicant)

# What does LIME say?
lime_explanation = explainer.explain_instance(applicant, model.predict_proba)

# What does SHAP say?
shap_explanation = shap_explainer.shap_values(applicant)

# Do they agree?
lime_important_features = set([f for f, _ in lime_explanation.as_list()[:5]])
shap_important_features = set(np.argsort(np.abs(shap_explanation[0]))[::-1][:5])

agreement_rate = len(lime_important_features & shap_important_features) / 5
print(f"Feature agreement between LIME and SHAP: {agreement_rate:.0%}")

# If agreement is low (< 50%), explanations are unreliable
# Investigate: Is the model learning spurious correlations?

# Step 3: Validate with domain experts
# Show the explanation to a loan officer. Do they agree it makes sense?
# If domain experts disagree with the explanation, the explanation is wrong.

Key lesson: Explanations can be plausible but false. Always:

1. Compare multiple explanation methods (SHAP vs LIME)
2. Validate with domain experts
3. Test edge cases (applicants just below/above decision boundary)
4. Monitor for shifts in explanation patterns (could indicate data drift)

Explainability Method Trade-off Summary

| Criterion | SHAP | LIME | Permutation | When to Use | |-----------|------|------|-------------|------------| | Latency per explanation | 1-10s (Kernel), 100ms (Tree) | 100-500ms | Batch only | Production: LIME or cache SHAP | | Interpretability | Marginal contribution (game theory) | Local linear model | % importance loss | Stakeholders prefer LIME simplicity | | Theory quality | Principled (Shapley values) | Heuristic | Empirical | Regulators prefer SHAP rigor | | Feature interactions | ✓ Yes | ✓ Weak | ✓ Yes | Complex models: SHAP > LIME | | Implementation complexity | Medium (many hyperparameters) | Low (usually default works) | Low | Development speed: LIME | | Cost to explain 1000 predictions | $1-50 (compute) | $0.10-0.50 | $5-20 | Budget-critical: Permutation |

Practical rule: Use LIME for real-time explanations, SHAP for audits and fairness analysis, permutation importance for model validation.

Explanation Deployment Architectures

| Architecture | Use Case | Cost | Latency | Flexibility | |-------------|----------|------|---------|------------| | Cache pre-computed SHAP | Fraud/lending (known decision types) | $100-500/mo | 1ms (lookup) | Low (only pre-computed) | | LIME on-demand | Web API (interactive) | $50-200/mo | 200-500ms | High (any prediction) | | Batch SHAP (nightly) | Batch analytics, fairness audits | $20-100/mo | N/A (offline) | High (any prediction) | | Hybrid: Cache + LIME fallback | Production (most requests cached) | $200-800/mo | 1ms (cache), 300ms (miss) | Very High |

Choose architecture based on: frequency of unique predictions, latency SLA, budget, and regulatory requirements.

Practical Tips for Production Explanations

1. Explain the most important decisions: In a mortgage approval system, you explain denials but not approvals (too many to explain, and users don't need explanation to accept good news). Allocate your explanation budget to high-stakes decisions.

2. Cache explanations: Computing SHAP for every prediction is slow. Pre-compute and cache explanations for common prediction types, regenerate on demand for edge cases.

3. Combine global and local: Show a user both global feature importance ("Income is most predictive of approvals overall") and their local explanation ("In your case, your credit score mattered most").

4. Validate explanations on real decisions: If your explainability method says feature X drove a decision, but domain experts know it didn't, your explanation method is wrong. Debug and retune.

5. Update explanations when you retrain: Old explanations on a new model are misleading. Recompute after each model update.

Common Mistake: Over-Interpreting Explanations

A customer denied a loan sees a LIME explanation: "Income <= 30k increased denial probability by +0.45". They assume this means the model is unfairly penalizing low income. But the explanation only says: in the local region around this customer's data, the linear approximation found that income matters. It doesn't mean:

1. Income is the true causal reason (confounders?)
2. The model is violating fair lending rules (many features can cause the same outcome)
3. Income should be removed (you might be legally or strategically required to use it)

Explanations are post-hoc interpretations of learned patterns, not proofs of causation or injustice. Always dig deeper with statistical testing, subgroup analysis, and domain expertise before drawing conclusions.

The lesson: Explainability is a debugging and communication tool, not a substitute for rigorous model validation and fairness analysis. Use explanations to identify hypotheses ("Maybe the model is biased against low-income applicants"), then test those hypotheses rigorously with fairness audits (see lesson 24).