Worked examples — Feature importance from trees

This page is the drill-ground for Feature importance from trees. The parent note built the machinery — the formula for Mean Decrease in Impurity (MDI). Here we throw every kind of situation at that machinery and grind out full answers, so you never meet a case you haven't already seen solved.

Before we compute anything, let's recall the one formula everything rests on. If it looks unfamiliar, go back to the parent — but here it is in plain words.

Every symbol above is a count or a fraction of counts — no calculus, no trig. If you can count marbles, you can do this whole page.

One more symbol we lean on constantly: $p_k$. When we write the Gini impurity $i(t)=1-{\sum }_k{p}_k^2$, the symbol $p_k$ means ==the fraction of the samples at node $t$ that belong to class $k$== — i.e. (count of class-$k$ samples in the node) ÷ (total samples in the node). For a node with $4$ positives and $6$ negatives, $p_{\text{pos}}=4/10=0.4$ and $p_{\text{neg}}=6/10=0.6$. Each $p_k$ is between $0$ and $1$, and across all classes the $p_k$ add up to $1$. The sum ${\sum }_k$ just runs over every class label present.

---

The scenario matrix

A tree-importance problem can only differ along a few axes. Here is the complete list of "case classes" — every worked example below is tagged with which cell it lands in.

#	Case class	What's different	Covered by
A	Root split, classification (Gini)	weight $=1$, cleanest baseline	Ex 1
B	Deep split, small arrival weight	weight $<1$, tests $N_t/N$	Ex 2
C	Same feature used twice in one tree	must sum two nodes	Ex 3
D	Regression tree (variance impurity)	$y$ values, not classes	Ex 4
E	Degenerate split — no cleanup	$\Delta i=0$, useless feature	Ex 5
F	Already-pure parent	$i(t)=0$, split can't help	Ex 6
G	Forest: average + normalize over trees	many trees → one ranking	Ex 7
H	High-cardinality trap (word problem)	MDI lies, needs the fix	Ex 8
I	Correlated twins share importance	one feature steals the credit	Ex 9

Cases A–D are the "signs and quadrants" — the ordinary geometry of the method. E and F are the degenerate / zero inputs (the equivalent of "what if the vector has length 0?"). G is the limiting/aggregation case. H and I are the real-world word problem and the exam twist where the naive number misleads.

The figure below is the picture behind Ex 1. What it encodes: the blue box at the top is the parent node with $5$ positives and $5$ negatives and impurity $0.50$. Two coloured arrows fan downward: the green arrow (labelled $4/10$) carries $4$ samples to a perfectly pure left child (green box, impurity $0.00$); the orange arrow (labelled $6/10$) carries $6$ samples to a mostly-clean right child (orange box, impurity $0.278$). The arrow labels $4/10$ and $6/10$ are the child weights $N_{t_L}/N_t$ and $N_{t_R}/N_t$. The gray line beneath shows the weighted-child arithmetic and the red banner at the very bottom is the final $\Delta i=0.333$ — read the figure now, then follow the arithmetic underneath it.

---

Ex 1 — Case A: root split, Gini classification

Forecast: guess first — will the importance be closer to $0$, to $0.3$, or to $0.5$? (The left child is perfectly clean, so cleanup should be large-ish.) The blue parent box in the figure above is where we start.

1. Parent impurity. Why this step? We need the baseline mess before we can measure how much we cleaned. Gini $=1-{\sum }_k{p}_k^2$ with $p_{\text{pos}}=p_{\text{neg}}=0.5$ (a perfect $50/50$ mix): $i(\text{root})=1-(0.5^2+0.5^2)=1-0.5=\mathrm{0.5.}$ This is the maximum possible Gini for two classes — a totally mixed node.

2. Child impurities. Why? We must see how pure each side became — the two smaller boxes in the figure. For the left, $p_{\text{pos}}=1,p_{\text{neg}}=0$; for the right, $p_{\text{pos}}=1/6,p_{\text{neg}}=5/6$: $i(L)=1-(1^2+0^2)=0(\text{perfectly pure}),$ $i(R)=1-\left({\frac{1}{6}}^2+{\frac{5}{6}}^2\right)=1-(0.02778+0.69444)=\mathrm{0.27778.}$

3. Weighted child impurity. Why? The right child holds more samples, so its (worse) purity should count more. Weight each child by its share of the parent's $10$ samples — the $4/10$ and $6/10$ labels on the arrows: $\frac{4}{10}(0)+\frac{6}{10}(0.27778)=\mathrm{0.16667.}$

4. Impurity decrease. Why? This is the actual cleanup: baseline minus leftover mess (the red banner): $\Delta i=0.5-0.16667=\mathrm{0.33333.}$

5. Arrival weight. Why? The root sees all data, so $N_t/N=10/10=1$; nothing shrinks. $\text{Imp}(X_1)=1\times 0.33333=\boxed{0.33333}.$

Verify: cleanup can't exceed the starting mess: $0.333<0.5$ ✓. The left child is pure so all its "credit" is real. Sanity: a useless split would give $\Delta i=0$; we got a big positive number, matching the strongly separating right side.

---

Ex 2 — Case B: deep split, small arrival weight

Forecast: its local cleanup ($0.4$) is bigger than $X_1$'s ($0.333$). Does that make $X_2$ more important? Guess yes/no.

1. Apply the arrival weight. Why this step? A split that only $3$ samples ever reach cannot matter as much as one touching everyone — even if locally it looks impressive. Multiply by $N_t/N=3/10$: $\text{Imp}(X_2)=\frac{3}{10}\times 0.4=\mathrm{0.12.}$

2. Rank against $X_1$. Why? Importance is about total contribution, not local flashiness. $\text{Imp}(X_1)=0.33333>\text{Imp}(X_2)=\mathrm{0.12.}$ So $X_1$ wins despite the smaller local drop — this is the entire reason $N_t/N$ exists.

3. Normalize. Why? We usually report shares that sum to $1$. Let $S$ be the total of all raw importances, $S=0.33333+0.12=0.45333$; dividing each importance by $S$ gives its share: $\widehat{\text{Imp}}(X_1)=\frac{0.33333}{0.45333}=0.73529,\widehat{\text{Imp}}(X_2)=\frac{0.12}{0.45333}=\mathrm{0.26471.}$

Verify: normalized shares sum to $1$: $0.73529+0.26471=1$ ✓. The bigger raw importance stays the bigger share, as it must.

---

Ex 3 — Case C: same feature used twice, sum the nodes

Forecast: one term or two? (Re-read the definition: we sum over every node splitting on $f$.)

1. Compute each node's contribution. Why this step? The formula is a sum "${\sum }_{t:\text{split}(t)=f}$" — a feature earns credit every time it's used. $\text{node }A:1.0\times 0.20=0.20,\text{node }B:0.30\times 0.50=\mathrm{0.15.}$

2. Sum them. Why? Total impurity work done by $X_3$ across the whole tree. $\text{Imp}(X_3)=0.20+0.15=\mathrm{0.35.}$

Verify: if we'd forgotten node $B$ we'd report $0.20$ — an undercount. Since $X_3$ is the only feature, its normalized importance is $0.35/0.35=1.0$ ✓ (everything must sum to $1$ with one feature).

---

Ex 4 — Case D: regression tree (variance impurity)

First we must define the impurity we'll use, since a regression node has no classes and Gini does not apply.

Forecast: the two children each contain values close together — big cleanup expected? Guess the $\Delta i$.

1. Parent impurity = variance about the mean. Why this step? In regression there are no classes; "mess" means "spread". Apply the definition above. Mean ${\bar{y}}_t=\frac{2+4+6+8}{4}=5$: $i(\text{parent})=\frac{1}{4}[(2-5{)}^2+(4-5{)}^2+(6-5{)}^2+(8-5{)}^2]=\frac{1}{4}(9+1+1+9)=5.$

2. Child variances. Why? Same measure, now for the tighter groups. Left mean $=3$, right mean $=7$: $i(L)=\frac{1}{2}[(2-3{)}^2+(4-3{)}^2]=\frac{1}{2}(1+1)=1,i(R)=\frac{1}{2}[(6-7{)}^2+(8-7{)}^2]=1.$

3. Weighted child impurity. Why? Equal-size children ($2$ and $2$), so equal weights $\frac{1}{2}$: $\frac{1}{2}(1)+\frac{1}{2}(1)=1.$

4. Impurity decrease. Why? Cleanup = baseline spread minus leftover spread: $\Delta i=5-1=4.$ Since arrival weight is $1$, the contribution is $1\times 4=4$.

Verify: children are much tighter than the parent, so a large drop is expected — and $4$ is $80\%$ of the original $5$ ✓. The feature that produced this split is separating $y$ extremely well.

---

Ex 5 — Case E: degenerate split, no cleanup

Forecast: the children look exactly like the parent. What cleanup can there be?

1. Parent impurity. Why? Baseline. $p_{\text{pos}}=p_{\text{neg}}=\frac{1}{2}$: $i(\text{parent})=1-(0.5^2+0.5^2)=\mathrm{0.5.}$

2. Child impurities. Why? Check if either side got purer. Both children are $50/50$, so again $p_{\text{pos}}=p_{\text{neg}}=\frac{1}{2}$: $i(L)=i(R)=1-(0.5^2+0.5^2)=\mathrm{0.5.}$

3. Weighted child impurity + decrease. Why? Combine and subtract: $\frac{4}{8}(0.5)+\frac{4}{8}(0.5)=0.5,\Delta i=0.5-0.5=0.$

4. Importance contribution. Why? $1\times 0=0$.

$X_4$ earns nothing here — a real greedy tree would never choose this split. This is the degenerate zero-input case: a feature can be present yet contribute $0$.

Verify: $\Delta i=0$ is the floor — impurity decrease is never negative (the greedy algorithm never picks a worsening split). We hit exactly the boundary ✓.

---

Ex 6 — Case F: already-pure parent

Forecast: the parent is already perfect. Can splitting a pure node clean anything?

1. Parent impurity. Why? Baseline. $p_{\text{pos}}=1$: $i(\text{parent})=1-(1^2+0^2)=0.$

2. Child impurities. Why? Both children are still all-positive ($p_{\text{pos}}=1$ each): $i(L)=1-(1^2)=0,i(R)=1-(1^2)=0.$

3. Impurity decrease. Why? $0-(\frac{3}{5}\cdot 0+\frac{2}{5}\cdot 0)=0.$

The importance contribution is $0$. A pure node has nothing to clean, so no feature can earn credit there. This is why trees stop splitting pure nodes: there's no impurity left to sell.

Verify: $\Delta i=0$, and it cannot go negative from a $0$ baseline (children $\ge 0$ always). Boundary respected ✓.

---

Ex 7 — Case G: forest average and normalize (limiting/aggregation)

Forecast: feature $A$ is consistently high; $C$ is nearly absent. Which normalized share is largest?

1. Average each row over the $T=3$ trees. Why this step? One tree is noisy; averaging is how Random Forests turn many unstable importances into a stable ranking (a small variance-reduction argument): $\bar{A}=\frac{0.30+0.20+0.40}{3}=0.30,\bar{B}=\frac{0.10+0.30+0.20}{3}=0.20,\bar{C}=\frac{0.00+0.10+0.00}{3}=\mathrm{0.03333.}$

2. Sum the averages. Why? To rescale so importances sum to $1$ (the usual reported form). Let $S$ denote the total of the three averaged importances: $S=\bar{A}+\bar{B}+\bar{C}=0.30+0.20+0.03333=\mathrm{0.53333.}$

3. Normalize. Why? Shares are easier to compare across models. Divide each averaged importance by $S$: $\hat{A}=\frac{0.30}{0.53333}=0.5625,\hat{B}=\frac{0.20}{0.53333}=0.375,\hat{C}=\frac{0.03333}{0.53333}=\mathrm{0.0625.}$

The bar chart below shows these two views side by side. What it encodes: three feature labels $A,B,C$ sit on the horizontal axis; the vertical axis is importance. For each feature there are two bars — the blue bar is the per-tree average from step 1 ($0.30$, $0.20$, $0.033$) and the orange bar is the normalized share from step 3 ($0.5625$, $0.375$, $0.0625$), each printed above its bar. The key visual message: the heights differ (normalizing rescaled everything) but the order blue-tall→short is identical to orange-tall→short, so the ranking $A>B>C$ is untouched.

Verify: shares sum to $1$: $0.5625+0.375+0.0625=1$ ✓. Ranking $A>B>C$ matches the raw averages ✓.

---

Ex 8 — Case H: high-cardinality trap (word problem)

Forecast: an ID is basically a name tag. Can a name tag really be the top predictor?

1. Name the mechanism. Why this step? We must explain why MDI over-credits customer_id. A split point is a threshold the tree can test. customer_id has $10,000$ distinct values, so it offers up to $9,999$ possible split thresholds; income_bracket offers at most $4$. More candidate splits = more chances to stumble onto a partition that happens to reduce impurity — even a partition that only isolates a few noisy training rows. Because MDI simply sums whatever impurity drop a feature achieved, this "many-tries advantage" inflates the score. This is the formal statement of Mistake 1 in the parent: MDI is biased toward high-cardinality / continuous features.

2. Predict what happens on new data. Why? To see the score is fake, ask whether it generalizes. An ID memorized on training rows carries no signal for a customer the model has never seen — its true out-of-sample predictive value is ~$0$. So the high training MDI is pure overfitting, an overfitting artifact (Mistake 4: MDI on training data overestimates).

3. Quantify how misleading the raw ranking looks. Why? To feel the size of the trap. Treating these two as the whole story, let $S=0.62+0.14=0.76$ and normalize: $\widehat{\text{id}}=\frac{0.62}{0.76}=0.81579,\widehat{\text{income}}=\frac{0.14}{0.76}=\mathrm{0.18421.}$ The ID looks responsible for $82\%$ of all importance — a wildly confident, wrong conclusion.

4. Apply the fix. Why? We need an honest measure. Use Permutation Importance: shuffle customer_id on a held-out set and measure the model's score drop. For a pure-memorization feature the held-out score barely moves → its importance collapses to ≈ $0$. Cross-check with SHAP Values for a per-prediction, less-biased attribution.

Answer: No. The high MDI is a cardinality + overfitting artifact, not causation. After permutation importance on held-out data, customer_id drops to ≈ $0$ and income_bracket emerges as the real (if modest) predictor.

Verify: normalized shares sum to $1$: $0.81579+0.18421=1$ ✓. The reasoning matches the parent's Mistake 1 and Mistake 4 ✓.

---

Ex 9 — Case I: correlated twins share importance (exam twist)

Forecast: the two features carry the same signal. Will MDI split the credit evenly? Guess before reading on.

1. Compute each feature's MDI. Why this step? MDI only credits a feature at nodes where it was actually used to split. $X_a$ was used once (root, weight $1$, $\Delta i=0.40$); $X_b$ was used zero times: $\text{Imp}(X_a)=1\times 0.40=0.40,\text{Imp}(X_b)=0\times (\cdots )=0.$

2. Interpret the gap. Why? To expose the trap. Despite identical predictive content, one twin reads $0.40$ and the other $0$. Once $X_a$ split the root, its twin $X_b$ became redundant — there was no impurity left for $X_b$ to reduce, so it earned nothing. A genuinely strong predictor is now hidden at importance $0$ (Mistake 2 in the parent).

3. State the honest combined importance. Why? No impurity work was lost — it was all attributed to $X_a$. So treating the twins as one unit, the pair did exactly: $\text{Imp}(\{X_a,X_b\})=\text{Imp}(X_a)+\text{Imp}(X_b)=0.40+0=\mathrm{0.40.}$

4. Recover it in practice. Why? To stop the credit-stealing. Cluster the correlated features and report importance per cluster, or use grouped Permutation Importance (shuffle both twins together and measure one combined score drop). Under grouping the pair correctly shows $\approx 0.40$ instead of the misleading "$0.40$ and $0$".

Answer: MDI gives $\text{Imp}(X_a)=0.40$, $\text{Imp}(X_b)=0$; the true combined importance of the pair is $0.40$, recoverable by clustering / grouped permutation.

Verify: the two raw numbers ($0.40$, $0$) are wildly unequal despite identical signal — exactly the correlated-feature failure mode ✓. Their sum $0.40$ equals the single-use contribution, so no impurity work went missing ✓.

---

Recall Which cell does each pitfall live in?

Ex 5 shows $\Delta i=0$ from a useless split ::: a feature can be present yet contribute nothing. Ex 6 shows a pure parent ::: impurity is already $0$, so no split can earn credit. Ex 8 is the high-cardinality trap ::: MDI over-credits many-valued features; fix with permutation/SHAP. Ex 9 is correlated twins ::: whichever is used first steals all the credit; the other reads $0$.

---

Connections

• Decision Trees — every $\Delta i$ here is the same quantity the tree maximized while growing.
• Gini vs Entropy — swap $i(t)$ and the numbers change but the procedure is identical.
• Random Forests — Ex 7 is exactly the forest averaging step.
• Gradient Boosting — same split-based importances (gain), same cardinality caveats.
• Permutation Importance — the fix invoked in Ex 8 and Ex 9.
• SHAP Values — per-prediction attribution that dodges Ex 8's bias.
• Bias-Variance Tradeoff — training-set MDI (Ex 8) overestimates via overfitting.