Feature attribution (SHAP, LIME): Question bank

Two words you must have straight before starting:

Attribution = a number handed to one feature for one specific prediction, saying how much that feature pushed the output.
Baseline / base value = the model's prediction when it has "seen nothing" — the average output over the data. Attributions explain the journey from baseline to this prediction, not the prediction itself.

True or false — justify

TRUE or FALSE: A feature with a large SHAP value is an important feature for the model overall.

FALSE. SHAP is instance-specific; a feature can dominate one prediction and be near-zero for another. For a global claim use 5.4.02-Permutation-importance instead.

TRUE or FALSE: If a feature has SHAP value zero for Bob, the model never uses that feature.

FALSE. Zero means it did not move Bob's prediction away from baseline — perhaps Bob's value equals the typical value. The feature may swing wildly for other applicants.

TRUE or FALSE: LIME and SHAP will always agree on the top feature.

FALSE. LIME fits a local linear surrogate by sampling perturbations; SHAP computes fair Shapley credit. They optimise different objectives, so their rankings can diverge, especially near non-linear kinks.

TRUE or FALSE: SHAP values for one instance must sum to the model's prediction.

TRUE — but precisely, they sum to

f (x) - ϕ_{0}

, the deviation from the base value. This is the efficiency axiom; add

ϕ_{0}

back to recover

f (x)

TRUE or FALSE: A positive SHAP value always means the feature increased the probability of the positive class.

FALSE in spirit. It means the feature pushed the model output (often a logit or margin) up. Whether that helps the "positive" class depends on how you defined the output being explained.

TRUE or FALSE: LIME is model-agnostic, so it needs the model's internal gradients.

FALSE. Model-agnostic means it only queries

f

as a black box (inputs → outputs). It never touches weights or gradients — that is exactly why it works on any model.

TRUE or FALSE: TreeSHAP gives approximate Shapley values, just faster than KernelSHAP.

FALSE. TreeSHAP computes exact Shapley values in polynomial time by exploiting tree structure; KernelSHAP is the approximate, model-agnostic one. See 3.5.04-Tree-ensembleinterpretation.

TRUE or FALSE: Running LIME twice on the same instance gives the same explanation.

FALSE (usually). LIME samples random perturbations, so with too few samples the fitted coefficients wobble between runs. Stability requires thousands of samples.

TRUE or FALSE: Feature attribution tells you the causal effect of a feature on the outcome.

FALSE. Attribution explains the model's behaviour, not the real world. A feature can be a proxy correlated with the target; SHAP credits it without any causal claim.

Spot the error

Claim: "I only generated 10 LIME perturbations because the linear fit is cheap." What is wrong?

The cost is not the fit — it is the variance. Ten samples make the local surrogate a noisy random guess; the paper recommends thousands (≈5,000+ tabular, 1,000+ images).

Claim: "For missing features in SHAP, I just set them to zero." Why is this wrong?

Zero is often not a neutral value — it can be an extreme input the model rarely sees. SHAP marginalises:

f_{S} (x_{S}) = E_{X_{- S}} [f (x_{S}, X_{- S})]

, replacing absent features with their typical distribution.

Claim: "LIME's superpixels are masked with black because black = absence." Why is gray usually better for ImageNet?

Black is a real, informative colour that can confuse the CNN. Gray sits near the ImageNet channel mean, acting as a truly neutral baseline that removes information without injecting new signal.

Claim: "Exact Shapley is fine, I'll just loop over all subsets." Where does this break?

The sum is over all

2^{d}

subsets. Even

d = 30

features gives over a billion evaluations — intractable. That is the whole reason KernelSHAP and TreeSHAP exist.

Claim: "My LIME explanation says feature

x_{3}

has weight

+ 2.4

, so increasing

x_{3}

by 1 raises the true prediction by 2.4." What is the subtle error?

The weight belongs to the local linear surrogate

g

, valid only in a small neighbourhood of

x

. Extrapolating it as a global slope of

f

ignores that

f

is non-linear away from

x

Claim: "Since SHAP is additive, the model must be a linear model." Why is this a confusion?

The additivity (

f (x) = ϕ_{0} + \sum_{i} ϕ_{i}

) is a property of the explanation, guaranteed by the efficiency axiom — not of

f

. Arbitrarily non-linear

f

still decomposes this way.

Why questions

Why do we weight perturbations by proximity

π_{x} (z) = exp (- d (x, z)^{2} / σ^{2})

in LIME?

Because we only want the surrogate to be faithful locally. Down-weighting far points lets the linear model bend to fit the neighbourhood of

x

without being dragged by distant, differently-behaving regions.

Why does the Shapley value average over all orderings of features rather than one?

A feature's marginal contribution depends on which features arrived before it (interactions). Averaging over every arrival order is the unique way to give fair credit that satisfies symmetry and efficiency simultaneously.

Why does SHAP add an L1-style / sparsity idea, and why does LIME use

Ω (g) = λ ∥ w ∥_{1}

LIME wants a human-readable explanation, so it penalises complexity to keep only a few non-zero weights. A dense 50-feature linear model is technically faithful but not interpretable.

Why can't we just read off global feature importance and call it a local explanation?

Global importance (e.g. permutation) averages over the whole dataset, hiding instance-level reversals. Attribution is local: it captures how this input's specific feature values drove this output.

Why does KernelSHAP "converge to true Shapley as samples → ∞" while LIME does not converge to Shapley?

KernelSHAP uses the specific Shapley kernel weights in its weighted regression, so its limit is provably the Shapley value. LIME uses proximity weights with a different goal, so its limit is a locally faithful surrogate, not Shapley credit.

Why are attention weights not a drop-in substitute for feature attribution?

Attention shows where the model looked, but high attention does not guarantee that token caused the output. Attribution measures marginal effect on the prediction, which attention alone cannot certify.

Edge cases

Edge case: two features are perfectly correlated (duplicate columns). How does SHAP split their credit?

By the symmetry axiom, two features contributing identically in every coalition receive equal Shapley values — the credit is split evenly between them, not doubled.

Edge case: a feature that never changes the prediction in any coalition. What is its SHAP value?

Exactly zero, guaranteed by the dummy axiom. A feature the model genuinely ignores gets no credit under any ordering.

Edge case: the instance

x

equals the baseline (all features at their average). What do the attributions look like?

They collapse toward zero — there is no deviation from

ϕ_{0}

to explain, so

\sum_{i} ϕ_{i} \approx 0

. Attributions explain a journey, and here there is no journey.

Edge case: LIME's neighbourhood width

σ

set very large. What happens to the explanation?

The surrogate becomes a global linear fit, losing local faithfulness — it starts describing average behaviour of

f

, which may badly mismatch

f

near

x

Edge case: LIME's

σ

set very small. What breaks now?

Almost all perturbations get near-zero weight, so the effective sample size shrinks and the linear fit becomes unstable and noisy — the same variance problem as too few samples.

Edge case: the model output being explained is a probability vs a logit. Why does the choice matter for SHAP?

Shapley additivity holds in whatever space you explain. Probabilities are squashed and non-linear, so attributions in logit space and probability space differ; always state which space

ϕ_{i}

lives in.

Edge case: you want to know "what minimal change flips Bob's decision," not "what drove it." Is attribution the right tool?

No — that is a counterfactual question. Attribution decomposes the current prediction; counterfactuals search for the nearest input with a different outcome.

Recall Fast self-test

Attribution is local or global? ::: Local — one feature, one instance. SHAP values sum to what? ::: $f (x) - ϕ_{0}$ , the deviation from the base value. LIME needs gradients? ::: No — pure black-box queries. TreeSHAP is exact or approximate? ::: Exact, in polynomial time for trees. Zero SHAP means the model ignores the feature? ::: No — only that it didn't move this prediction from baseline.