Feature attribution (SHAP, LIME): Visual walkthrough

We keep ONE running example the whole way:

Step 1 — What does "a feature contributes" even mean?

WHAT. Before any maths, we need to say what it means for a feature to "help" a prediction. The idea: a feature contributes by how much the prediction changes when we reveal it.

WHY. We cannot look at a feature in isolation — the model is tangled. But we can compare "prediction knowing $C$ " against "prediction knowing nothing." That difference is a marginal gain — the amount the number moved because $C$ walked into the room.

PICTURE. Look at the figure: a horizontal number line from the base value $0.20$ up to the full prediction $0.72$ . Revealing credit alone slides the marker up. The length of that orange arrow is credit's marginal gain, here $0.35 - 0.20 = + 0.15$ .

Step 2 — The gain depends on WHO arrived first

WHAT. We reveal credit's gain again — but this time income and debt are already known. The number credit adds is different now.

WHY. The model is non-linear: features interact. If income already told the model "this person is rich," then learning the credit score adds less new information. So credit's contribution is not one number — it changes with the order features arrive.

PICTURE. Two stacked number lines. Top: credit arrives first, big orange jump. Bottom: credit arrives last, tiny orange jump. Same feature, same applicant — different credit depending on context.

Step 3 — What is $f_{S}$ ? Filling the empty chairs

WHAT. We've used $f_{S}$ ("predict knowing only the features in $S$ "). But a trained model demands all its inputs. How do we run it when some features are unknown?

WHY. We can't leave a feature blank. Setting it to $0$ would be a lie (income $0$ is a meaningful, extreme value). Instead we ask: "averaged over all the incomes real people have, what does the model say?" That's an expected value — the honest stand-in for "unknown."

PICTURE. A chair diagram: known features sit fixed in their chairs; each unknown feature's chair is filled by many faint sampled values, and we average the predictions.

Step 4 — List every ordering (there are $d!$ of them)

WHAT. With 3 features, we write out every order they could arrive in. Three features have $3! = 3 \times 2 \times 1 = 6$ orderings.

WHY. Step 2 said credit's gain depends on order. To be fair to no ordering, we consider all of them equally. The factorial $d!$ (read " $d$ factorial", $= d \cdot (d - 1) \dots 1$ ) counts arrangements: 3 choices for first, 2 for second, 1 for last.

PICTURE. All 6 permutations of ${C, I, D}$ drawn as 6 rows of arriving players, with credit ( $C$ ) highlighted and the set already present (its $S$ ) marked just before $C$ steps in.

For credit, tabulate the $S$ -before- $C$ across all 6 orderings:

| Ordering | Set $S$ before $C$ arrives | $∣ S ∣$ | |---|---|---| | $C, I, D$ | $\emptyset$ | 0 | | $C, D, I$ | $\emptyset$ | 0 | | $I, C, D$ | ${I}$ | 1 | | $D, C, I$ | ${D}$ | 1 | | $I, D, C$ | ${I, D}$ | 2 | | $D, I, C$ | ${I, D}$ | 2 |

\emptyset

the empty set — the room is empty,

C

arrives first.

Step 5 — Group the orderings: where the weights come from

WHAT. Notice the table repeats. $S = \emptyset$ shows up twice; $S = {I}$ once; $S = {D}$ once; $S = {I, D}$ twice. We don't need all 6 rows — we need each distinct $S$ weighted by how often it occurs.

WHY. Two orderings both give $S = \emptyset$ because after $C$ arrives first, the remaining two features ${I, D}$ can still be seated in $2! = 2$ ways — and those don't affect credit's gain. This is the birth of the combinatorial weight.

PICTURE. A bar chart: the 6 orderings collapsed into buckets by $S$ . Bucket heights $2, 1, 1, 2$ — sum $6$ .

Divide by the total $d!$ orderings to get a probability weight: $w (S) = \frac{∣ S ∣ ! ( d - ∣ S ∣ - 1 )!}{d !}$

For $d = 3$ : $w (\emptyset) = \frac{0 ! 2 !}{3 !} = \frac{2}{6} = \frac{1}{3}$ , and $w ({I, D}) = \frac{2 ! 0 !}{3 !} = \frac{1}{3}$ , and each single-element $S$ gets $\frac{1 ! 1 !}{3 !} = \frac{1}{6}$ .

Step 6 — Assemble the Shapley value

WHAT. Multiply each marginal gain by its weight and sum. That weighted average is feature $i$ 's fair credit.

WHY. Every ordering counted exactly once, no ordering favoured — that is fairness by construction. The weights are not arbitrary; they are simply "what fraction of orderings put this $S$ before $i$ ."

PICTURE. A funnel: all six marginal gains flow in on the left, get scaled by their weight bars, and merge into one number $ϕ_{C}$ on the right.

Plug in our numbers. Using the parent's marginal gains for credit across its distinct $S$ -groups (weights $\frac{1}{3}, \frac{1}{6}, \frac{1}{6}, \frac{1}{3}$ ), a representative computation gives: $ϕ_{C} = \frac{1}{3} (0.15) + \frac{1}{6} (0.10) + \frac{1}{6} (0.12) + \frac{1}{3} (0.08) = 0.113 \overline{3}$

Step 7 — Edge & degenerate cases (never hit a wall)

WHAT. We test the formula at its boundaries.

WHY. A fair rule must behave sanely at extremes, or we don't trust it.

PICTURE. Three mini-panels: a dummy feature (flat line, zero credit), two twin features (equal bars), and the sum-check bar reaching exactly $f (x)$ .

The one-picture summary

This single figure stacks the whole derivation: base value $ϕ_{0}$ at the bottom, each feature's Shapley value $ϕ_{i}$ as a colored block, arriving in fair-averaged fashion, landing precisely on $f (x) = 0.72$ . This "waterfall" is exactly what a SHAP plot shows in practice.

Recall Feynman retelling — say it in plain words

Picture features as people walking into a room one at a time. Each time someone new walks in, the model's prediction jumps a little — that jump is what they added given who was already there. The trouble: the size of your jump depends on who came before you, so there's no single "your contribution." Shapley's fix: imagine every possible order people could walk in, measure each person's jump in each order, and average. Because some orders share the same "already-seated crowd," we don't recompute them — we just count how many orders share it (that's the factorial weight $∣ S ∣! (d - ∣ S ∣ - 1)! / d!$ ). The average is each person's fair share, and — magically — the shares add up to the exact prediction minus the base value, with nothing left over. That "adds up exactly" property is the efficiency axiom, and it's why a SHAP waterfall always lands cleanly on the real output.

Recall Quick self-check

Why must we average over all orderings? ::: Because a feature's marginal gain depends on which features arrived before it (interactions); averaging over every order is the only order-fair choice. Where do the factorial weights come from? ::: They count how many of the $d!$ orderings place a given set $S$ before feature $i$ : $∣ S ∣!$ ways to arrange those before, $(d - ∣ S ∣ - 1)!$ ways for those after. What does the efficiency axiom guarantee? ::: The Shapley values sum exactly to $f (x) - ϕ_{0}$ , so the explanation has no unexplained residual. Why replace missing features with an expectation, not zero? ::: Zero is a real, often extreme feature value; the expectation is the honest "typical" stand-in matching the base value.

Parent: Feature attribution (SHAP, LIME)
6.301-Global-vs-local-interpretability — this is a local explanation
5.4.02-Permutation-importance — a global alternative that averages, not attributes per-instance
3.5.04-Tree-ensembleinterpretation — where TreeSHAP computes these values exactly and fast
6.3.03-Counterfactual-explanations — the "what would flip it?" cousin
6.2.03-Attention-mechanisms — another built-in attribution signal