Feature attribution (SHAP, LIME): Worked examples

This is the hands-on companion to Feature attribution (SHAP, LIME). The parent note built the ideas: LIME's local linear fit and SHAP's fair Shapley credit. Here we grind through every case class you can meet — the friendly ones and the traps.

Before we start, one promise from the contract: no symbol appears before it is earned. We reuse only three things from the parent, and we restate each in plain words the first time it shows up:

Recall The three symbols we carry over

$f (x)$ ::: the model's prediction for one input $x$ (a single number, e.g. a probability like $0.72$ ).
$ϕ_{0}$ ::: the base value — the average prediction if you knew nothing about the instance. Written $ϕ_{0} = E [f (X)]$ , read "expected value of $f$ over all inputs", i.e. the plain average output.
$ϕ_{i}$ ::: the Shapley value of feature $i$ — the fair amount of the gap $f (x) - ϕ_{0}$ that we hand to feature $i$ .

The scenario matrix

Every attribution problem falls into one of these cells. The examples below are labelled by cell letter so you can see the whole grid is covered.

Cell	Case class	What makes it tricky	Example
A	Positive push (feature raises output)	sign of $ϕ_{i}$ is $+$	Ex 1
B	Negative push (feature lowers output)	sign of $ϕ_{i}$ is $-$	Ex 2
C	Efficiency check (do the pieces sum?)	$\sum ϕ_{i}$ must equal $f (x) - ϕ_{0}$ exactly	Ex 3
D	Dummy feature (zero contribution)	degenerate: $ϕ_{i} = 0$	Ex 4
E	Interaction / order dependence	marginal gain differs per ordering	Ex 5
F	LIME local linear fit (tabular)	weighted regression, sign of weights	Ex 6
G	LIME instability (limiting behaviour)	too-few-samples variance blow-up	Ex 7
H	Real-world word problem	translate a story into $ϕ$ 's	Ex 8
I	Exam twist (base value confusion)	mixing $ϕ_{0}$ with a feature's $ϕ_{i}$	Ex 9

We work SHAP first (cells A–E, I), then LIME (F, G), then a story (H).

Look at the bar picture above: this is the mental model for every SHAP example. Start at the base value $ϕ_{0}$ (grey), then each feature's $ϕ_{i}$ is an arrow pushing the running total right (positive, blue) or left (negative, orange), and you must land exactly on $f (x)$ .

Warm-up: a tiny 2-feature model we can compute by hand

To do Shapley values exactly (not sampled), we need a model small enough to write down. Let features be $x_{1}, x_{2}$ , each either $0$ or $1$ . Define the model on all four possible inputs:

What is $ϕ_{0}$ ? The average over all four inputs (each equally likely):

ϕ_{0} = \frac{1}{4} (2 + 5 + 4 + 9) = 5.

Why the average? $ϕ_{0}$ is "the prediction before you look at any feature" — so you average over everything you don't yet know.

We will attribute the gap $f (x) - ϕ_{0} = 9 - 5 = 4$ between features $1$ and $2$ .

SHAP examples

Worked example Ex 1 — Cell A: a feature that pushes UP

Statement. In the toy game, compute $ϕ_{1}$ for instance $(1, 1)$ . Feature 1 flips from its "unknown" state to the value $1$ .

Forecast: guess — do you think $ϕ_{1}$ comes out positive or negative? (Setting $x_{1} = 1$ raised outputs in the table, so...)

List the coalitions $S$ not containing feature 1. With $d = 2$ , feature 1 can be preceded by either $S = {}$ (empty) or $S = {2}$ . Why this step? Shapley averages feature 1's marginal gain over every set of features that could have "arrived before" it.
Marginal gain when $S = {}$ . Adding feature 1 to nothing: $f_{{1}} (x_{1} = 1) - f_{{}}$ . Here $f_{{}} = ϕ_{0} = 5$ (know nothing), and $f_{{1}} (1)$ = average over the unknown $x_{2}$ : $\frac{1}{2} (f (1, 0) + f (1, 1)) = \frac{1}{2} (5 + 9) = 7$ . Gain $= 7 - 5 = 2$ . Why this step? "Feature known, the rest unknown" means average over the rest — that is $f_{S}$ .
Marginal gain when $S = {2}$ . Now $x_{2} = 1$ is already known. $f_{{1, 2}} = f (1, 1) = 9$ . And $f_{{2}} (x_{2} = 1) = \frac{1}{2} (f (0, 1) + f (1, 1)) = \frac{1}{2} (4 + 9) = 6.5$ . Gain $= 9 - 6.5 = 2.5$ . Why this step? Same recipe, but now feature 2 is fixed, so we only average over the still-unknown feature 1's complement — none left, so it's exact.
Combinatorial weights. For $d = 2$ : weight of $∣ S ∣ = 0$ is $\frac{0 ! ( 2 - 0 - 1 )!}{2 !} = \frac{1}{2}$ ; weight of $∣ S ∣ = 1$ is $\frac{1 ! 0 !}{2 !} = \frac{1}{2}$ . Why this step? Half the orderings put feature 1 first, half put it second — each configuration weighs $\frac{1}{2}$ .
Combine. $ϕ_{1} = \frac{1}{2} (2) + \frac{1}{2} (2.5) = 2.25$ .

Verify: positive, as forecast — feature 1 pushes the prediction up by $+ 2.25$ . Units: same as $f$ (prediction points). ✓

The waterfall above visualises Ex 3: base $5 \to + 2.25 \to + 1.75 \to 9$ . If the last bar doesn't touch $f (x)$ , you have a bug.

Worked example Ex 5 — Cell E: interaction, why ordering matters

Statement. An AND model: $h (x_{1}, x_{2}) = 1$ only if both are $1$ , else $0$ . Explain $x = (1, 1)$ , $h = 1$ . Show the marginal gain of feature 1 depends on order, then compute $ϕ_{1}$ .

Forecast: neither feature alone does anything — you need both. So do they split the credit evenly?

First $ϕ_{0} = \frac{1}{4} (0 + 0 + 0 + 1) = 0.25$ .

Feature 1 arriving first ( $S = {}$ ). $h_{{1}} (x_{1} = 1) = \frac{1}{2} (h (1, 0) + h (1, 1)) = \frac{1}{2} (0 + 1) = 0.5$ . Base $0.25$ . Gain $= 0.25$ .
Feature 1 arriving second ( $S = {2}$ ). $x_{2} = 1$ known: $h_{{1, 2}} = h (1, 1) = 1$ ; $h_{{2}} (x_{2} = 1) = \frac{1}{2} (h (0, 1) + h (1, 1)) = \frac{1}{2} (0 + 1) = 0.5$ . Gain $= 0.5$ . Why this step? The gains ( $0.25$ vs $0.5$ ) differ — that difference IS the interaction. Shapley's averaging is what tames it.
Weighted average. $ϕ_{1} = \frac{1}{2} (0.25) + \frac{1}{2} (0.5) = 0.375$ . By symmetry $ϕ_{2} = 0.375$ .
Efficiency check. $ϕ_{1} + ϕ_{2} = 0.75 = h (x) - ϕ_{0} = 1 - 0.25 = 0.75$ . ✓

Verify: credit split evenly ( $0.375$ each) because features are symmetric, even though single marginal gains were unequal ( $0.25 \neq = 0.5$ ). That is exactly why we average over orderings. ✓

LIME examples

LIME does not use Shapley weights. Instead it fits a weighted linear model near $x$ . Symbols we earn now:

Worked example Ex 6 — Cell F: LIME weighted linear fit on 1 feature

Statement. Toy tabular explanation. We perturb one binary feature $z \in {0, 1}$ around $x$ (where $x$ 's feature $= 1$ ). We collect these (sample, model output $f$ , weight $π$ ):

$z$	$f (z)$	$π_{x} (z)$
1	0.9	1.0
1	0.8	1.0
0	0.3	0.5
0	0.2	0.5

Fit $g (z) = w_{0} + w_{1} z$ by weighted least squares. Find $w_{1}$ (the LIME attribution for the feature).

Forecast: turning the feature on ( $z : 0 \to 1$ ) raised $f$ from ~0.25 to ~0.85. So $w_{1}$ is roughly...?

Write weighted normal equations. Minimise $\sum_{z} π_{x} (z) [f (z) - w_{0} - w_{1} z]^{2}$ . Since $z$ is either 0 or 1, the fitted line just passes through the two weighted group means. Why this step? With a binary predictor, weighted regression reduces to "weighted mean of the $z = 1$ group" and "of the $z = 0$ group".
Weighted mean at $z = 1$ . $\frac{1.0 ( 0.9 ) + 1.0 ( 0.8 )}{1.0 + 1.0} = \frac{1.7}{2} = 0.85$ . So $w_{0} + w_{1} = 0.85$ .
Weighted mean at $z = 0$ . $\frac{0.5 ( 0.3 ) + 0.5 ( 0.2 )}{0.5 + 0.5} = \frac{0.25}{1} = 0.25$ . So $w_{0} = 0.25$ .
Solve. $w_{1} = 0.85 - 0.25 = 0.60$ .

Verify: $w_{1} = 0.60 > 0$ , matching the forecast: switching the feature on adds ~ $0.6$ to the prediction. Intercept $w_{0} = 0.25$ is the predicted output with the feature off. Units: prediction points per unit feature. ✓

Worked example Ex 7 — Cell G: LIME instability (limiting behaviour)

Statement. A colleague runs LIME with only $n = 4$ perturbations and gets $w_{1} = 0.60$ ; rerun gives $w_{1} = 0.15$ . Explain why, and what the estimate does as $n \to \infty$ .

Forecast: is LIME deterministic, or is that swing a bug?

LIME samples randomly. Each run draws a different random set of perturbations $z$ , so the fitted $w_{1}$ is a random variable. Its spread (variance) shrinks like $\propto 1/ n$ . Why this step? Fewer samples = the weighted regression is estimated from little data = high variance.
Quantify the swing. With $n = 4$ , standard deviation is large; two runs of $0.60$ and $0.15$ (difference $0.45$ ) are within normal noise, not a contradiction.
Limiting behaviour. As $n \to \infty$ , the sampling variance $\to 0$ and $w_{1}$ converges to a single stable value — the true local-linear coefficient. Why this step? This is the degenerate/limit cell of the matrix: the estimator is only trustworthy in the large-sample regime.

Verify: the fix is exactly what the parent note warns — use thousands of samples and check top-feature stability across reruns. A $0.45$ swing at $n = 4$ is expected, not alarming. ✓

Real-world & exam cells

Worked example Ex 8 — Cell H: loan word problem, end to end

Statement. A model outputs $P (default) = 0.72$ for Bob. Base value $ϕ_{0} = 0.20$ . SHAP returns: credit $ϕ_{cr} = + 0.10$ , income $ϕ_{inc} = + 0.05$ , debt $ϕ_{debt} = + 0.22$ , age $ϕ_{age} = + 0.15$ . (a) Do these obey efficiency? (b) Which single feature is most responsible, and does its sign mean "raises" or "lowers" default risk?

Forecast: the four $ϕ$ 's look positive. Should they sum to $0.72$ or to $0.72 - 0.20$ ?

Efficiency uses the gap, not $f (x)$ . Required: $\sum ϕ_{i} = f (x) - ϕ_{0} = 0.72 - 0.20 = 0.52$ . Why this step? Shapley credit explains the deviation from average, never the raw prediction.
Sum the pieces. $0.10 + 0.05 + 0.22 + 0.15 = 0.52$ . Matches. ✓
Rank magnitudes. Debt $0.22$ is largest, so debt-to-income is the top driver.
Read the sign. $+ 0.22$ means debt pushes the prediction up, i.e. raises default probability.

Verify: $0.10 + 0.05 + 0.22 + 0.15 = 0.52 = 0.72 - 0.20$ ; efficiency holds and debt (+) is Bob's biggest risk factor. If they'd summed to $0.72$ you'd know a $ϕ$ was wrong or $ϕ_{0}$ was misread. ✓

Worked example Ex 9 — Cell I: the exam twist (base-value trap)

Statement. "A student writes $\sum_{i} ϕ_{i} = f (x) = 0.72$ and concludes the SHAP output is valid." Spot the error, and state the correct identity.

Forecast: what did they forget?

Recall the true decomposition. $f (x) = ϕ_{0} + \sum_{i} ϕ_{i}$ , so $\sum_{i} ϕ_{i} = f (x) - ϕ_{0}$ , not $f (x)$ . Why this step? The base value $ϕ_{0}$ is the "you-know-nothing" starting point; the $ϕ_{i}$ only explain the journey from there.
Apply Bob's numbers. Correct sum $= 0.52$ , but the student's claim needs $0.72$ . Those differ by exactly $ϕ_{0} = 0.20$ . Why this step? The mismatch is precisely the forgotten base value — a dead giveaway.

Verify: $0.72 - 0.52 = 0.20 = ϕ_{0}$ . The student folded the base value into the features. Correct identity: $\sum_{i} ϕ_{i} = f (x) - ϕ_{0}$ . ✓

Wrap-up: cross-links

Global cousins of these local scores: global vs local interpretability and permutation importance.
TreeSHAP exploits structure explained in tree-ensemble interpretation.
Attribution's sibling that asks "what change flips the decision?" — counterfactual explanations.
Attention scores are a different kind of attribution — see attention mechanisms.

Recall Self-test

Why must SHAP $ϕ_{i}$ sum to $f (x) - ϕ_{0}$ and not to $f (x)$ ? ::: The $ϕ_{i}$ explain the deviation from the base value $ϕ_{0}$ ; the efficiency axiom gives $f (x) = ϕ_{0} + \sum_{i} ϕ_{i}$ . In the AND game (Ex 5), why do features get equal credit despite unequal marginal gains? ::: Symmetry: averaging feature 1's gain over orderings ( $0.25$ and $0.5$ ) yields the same $0.375$ as feature 2's, so symmetric players share equally. A LIME weight $π_{x} (z)$ at $d = 0$ equals what, and why exponential? ::: Exactly $1$ ; the exponential is smooth, always positive, and finite at $d = 0$ (unlike $1/ d$ ).