Foundations — Feature attribution (SHAP, LIME)

Before you can read Feature attribution (SHAP, LIME), you need to already see a handful of symbols and ideas. The parent page throws around $f$, $x$, ${\varphi }_i$, ${\pi }_x$, subsets $S$, expectations $\mathbb{E}$, and the word "baseline." This page builds each one from nothing, in an order where every symbol is earned before it is used.

1. The model as a machine: $f$ and $x$

Start with the most basic picture. A model is a machine with slots. You drop numbers into the slots, turn the crank, and one number comes out.

Figure s01 — the machine picture. Four coloured slot-boxes ($x_1,\dots ,x_4$) feed arrows into a central box labelled $f$; one arrow leaves it to a single output $f(x)=0.72$. Use it to see that many inputs collapse to one number — which is exactly the number attribution will later split back apart.

WHY we need this notation. Attribution asks "how much did slot $i$ matter?" You cannot ask that without a name for each slot ($x_i$), for the whole list ($x$), and for the machine ($f$). The subscript $i$ is just a house number so we can point at one feature.

2. Turning slots on and off: subsets $S$

Attribution keeps asking "what if this feature weren't there?" So we need a way to talk about which slots are switched on.

Figure s02 — switches as coalitions. Three rows of four circular switches (green = on, grey = off) show the empty coalition $\varnothing$, a partial coalition $\{1,3\}$, and the full set. Use it to see that a subset $S$ is just one on/off pattern, and that there are $2^d$ such patterns — the count that makes exact SHAP expensive.

WHY we need subsets. A feature's importance depends on its company. Credit score matters a lot when income is unknown, but less when income is already revealed. To measure this honestly we must try the feature in every possible group of companions — that is exactly what "for all subsets $S$" means in the Shapley formula.

3. The "hidden slot" problem: expectation $\mathbb{E}$

Machines demand a number in every slot. But when a feature is "hidden," what do we feed it? We cannot leave the slot empty and we should not feed a fake zero.

WHY averaging is the right "fill-in." To hide a feature, we replace it not with a single guess but with all the values it could plausibly take, blended by their frequency. If we hid "income," we ask: "averaging over the incomes typical of this population, what does the model say?" That blended answer is written with $\mathbb{E}$.

The honest way to write this fixes the known slots at their real values and averages only the hidden ones — conditioned on what we already know:

Let us decode every piece, since the parent uses this line without unpacking it:

Figure s03 — averaging fills the hidden slot. Two slots (credit, income) are fixed; several draws of the hidden slots (debt, age) — drawn consistently with the known values — each run through $f$ and the outputs are averaged into $f_S(x_S)$. Use it to see why a hidden feature becomes an average, not a zero.

4. Two anchor predictions: baseline ${\varphi }_0$ and full $f(x)$

Every attribution measures a journey between two numbers.

WHY a baseline at all? "This applicant has 72% risk" is meaningless without a reference. Attribution explains the deviation from average: the features must, together, account for the whole $+0.52$ climb from the crowd's average to Bob's number. That "together account for exactly the gap" is the efficiency promise on the parent page.

5. The credit slice: ${\varphi }_i$ and the marginal gain ${\Delta }_i(S)$

Now the star symbol.

The building block of ${\varphi }_i$ is one small measurement:

Figure s04 — the marginal-gain staircase. Predictions climb $0.20\to 0.35\to 0.40\to 0.58\to 0.72$ as credit, income, debt, age are switched on one at a time; each riser is labelled with its ${\Delta }_i(S)$. Use it to see that the four gains stack to fill the whole $+0.52$ gap.

WHY average over many $S$? From the loan example, adding "debt" gave $+0.18$ in one ordering. In a different ordering (different $S$ already on) it gives a different jump. A fair score cannot depend on the accident of ordering, so ${\varphi }_i$ is the average jump over all subsets $S$ that don't already contain $i$ — with combinatorial weights that count how many orderings produce each $S$. Written out fully:

6. "Local neighbourhood" and the weight ${\pi }_x$ (for LIME)

LIME needs one more idea: closeness. It builds a simple model that is accurate only near $x$.

WHY the exponential? We want a smooth dial that is $1$ at the centre and fades to $0$ as things get far — never negative, never jumping. $e^{-({\text{distance}}^2)}$ is the standard "soft bubble" shape (a bell) that does exactly this. It is the reason LIME "only cares about fidelity in the local region."

Now we can see where the weight actually enters: LIME fits a simple linear model $g$ by minimising a ${\pi }_x$-weighted squared error, so perturbations near $x$ dominate the fit:

How the foundations feed the topic

Both roads end at the same place: split $f(x)-{\varphi }_0$ into per-feature pushes. Related tools that reuse these ideas: 5.4.02-Permutation-importance (global cousin of attribution), 3.5.04-Tree-ensembleinterpretation (where TreeSHAP lives), 6.2.03-Attention-mechanisms (a different "what mattered" signal), and 6.3.03-Counterfactual-explanations (the "what would flip it" sibling).

Equipment checklist

Test yourself — say the answer aloud before revealing.