Feature importance from trees: Exercises — AI-ML deep dive

The figure above is the object every problem below lives inside: a parent bucket splitting into a left and right child. Keep it in your mind's eye.

Level 1 — Recognition

L1.1

For a binary classification node with class proportions $p_{1} = 0.5, p_{2} = 0.5$ , what is its Gini impurity, and what does that number mean in words?

Recall Solution

WHAT we do: plug into $i_{Gini} = 1 - \sum_{k} p_{k}^{2}$ . $i = 1 - (0. 5^{2} + 0. 5^{2}) = 1 - 0.5 = 0.5.$ WHAT it means: if you draw two points at random (with replacement) from this bucket, the chance they belong to different classes is $0.5$ . A 50/50 bucket is the most mixed a binary node can be — that is why $0.5$ is the maximum Gini for two classes.

L1.2

Which of these correctly names the quantity $\frac{N _{t}}{N}$ ? (a) impurity, (b) arrival weight, (c) impurity decrease, (d) Gini index.

Recall Solution

(b) arrival weight. It is the fraction of all data that reaches node $t$ . It has nothing to do with how mixed the node is (that's impurity) — it is purely a "how many people showed up" head-count.

L1.3

A leaf holds only class-A points. State its Gini impurity and its variance impurity (if we treated the labels as numbers all equal to $7$ ).

Recall Solution

Pure bucket → $p_{A} = 1$ . Gini $= 1 - 1^{2} = 0$ . For variance, every $y = 7$ so $\overset{y}{ˉ} = 7$ and $\frac{1}{N _{t}} \sum (y - \overset{y}{ˉ})^{2} = 0$ . Both impurities are $0$ — a pure node has nothing left to clean.

Level 2 — Application

L2.1

Root of a tree: 20 samples, 12 class-P and 8 class-N. Compute the root Gini impurity.

Recall Solution

$p_{P} = 12/20 = 0.6, p_{N} = 8/20 = 0.4$ . $i = 1 - (0. 6^{2} + 0. 4^{2}) = 1 - (0.36 + 0.16) = 1 - 0.52 = 0.48.$

L2.2

That root (20 samples) splits on feature $X_{1}$ into:

Left: 8 samples (6 P, 2 N)
Right: 12 samples (6 P, 6 N)

Compute $Δ i$ for this split, then $Imp (X_{1})$ (the split is at the root, so its arrival weight is $1$ ).

Recall Solution

Step 1 — child impurities (why: measure how clean each side got). Left: $p_{P} = 6/8 = 0.75, p_{N} = 0.25$ → $i_{L} = 1 - (0.5625 + 0.0625) = 0.375$ . Right: $p_{P} = 6/12 = 0.5, p_{N} = 0.5$ → $i_{R} = 1 - (0.25 + 0.25) = 0.5$ . Step 2 — weighted child impurity (why: bigger child counts more). $\frac{8}{20} (0.375) + \frac{12}{20} (0.5) = 0.15 + 0.30 = 0.45.$ Step 3 — decrease. $Δ i = 0.48 - 0.45 = 0.03$ . Step 4 — arrival weight $= 20/20 = 1$ , so $Imp (X_{1}) = 1 \times 0.03 = 0.03$ . The split barely helped — the right child is still 50/50, so this feature does little impurity work.

L2.3 (regression)

A regression node holds $y = {1, 3, 5, 7, 9}$ . Compute its variance impurity $i_{MSE}$ .

Recall Solution

$\overset{y}{ˉ} = (1 + 3 + 5 + 7 + 9) /5 = 5$ . Deviations squared: $16, 4, 0, 4, 16$ , sum $= 40$ . $i_{MSE} = \frac{40}{5} = 8.$

Level 3 — Analysis

L3.1

Feature $A$ splits the root (all $N = 100$ samples) with local $Δ i = 0.10$ . Feature $B$ splits a deeper node reached by only $25$ samples, with a larger local $Δ i = 0.30$ . Which feature gets more importance from these two splits, and why?

Recall Solution

WHAT we compute: apply the arrival weight to each. $Imp from A = \frac{100}{100} (0.10) = 0.10, Imp from B = \frac{25}{100} (0.30) = 0.075.$ Answer: $A$ (0.10) beats $B$ (0.075) even though $B$ 's local cleanup was three times bigger. WHY: $B$ 's split only touched a quarter of the data, so its cleanup is scaled down by $0.25$ . Importance rewards splits that clean a lot of people, not just clean a lot. This is the entire reason the $N_{t} / N$ weight exists — see Bias-Variance Tradeoff for why we distrust deep, low-sample splits.

L3.2

Normalise the two importances from L3.1 so they sum to $1$ .

Recall Solution

Total $= 0.10 + 0.075 = 0.175$ . $Imp (A) = \frac{0.10}{0.175} = 0.5714, Imp (B) = \frac{0.075}{0.175} = 0.4286.$ Check: $0.5714 + 0.4286 = 1$ ✓. Normalising lets you read each number as "fraction of total impurity work done."

L3.3

Two features $C$ and $D$ are perfectly correlated (identical columns). The tree happens to use $C$ at every relevant split and never uses $D$ . Predict the MDI of $C$ and of $D$ , and explain the danger.

Recall Solution

$Imp (C)$ is large; $Imp (D) \approx 0$ because $D$ never appears in a split, and MDI only sums over nodes that split on that feature. The danger: a naïve reader concludes " $D$ is useless" — but $D$ is just as predictive as $C$ ; it lost a coin-flip for who-gets-used. Fix: cluster correlated features, or use grouped Permutation Importance / SHAP Values.

Level 4 — Synthesis

L4.1 — Build a whole tree's importances

A dataset has $N = 16$ samples, 8 class-0 and 8 class-1. The fitted tree is:

Root (16 samples, 8/8): splits on feature $F_{1}$ into
- Node L (8 samples, 7 class-0, 1 class-1): splits on feature $F_{2}$ into
  - Leaf LL (4 samples, 4 class-0, 0 class-1)
  - Leaf LR (4 samples, 3 class-0, 1 class-1)
- Leaf R (8 samples, 1 class-0, 7 class-1): leaf.

Compute $Imp (F_{1})$ , $Imp (F_{2})$ , then the normalised importances.

Recall Solution

Root split (on $F_{1}$ ), arrival weight $= 16/16 = 1$ . $i (root) = 1 - (0. 5^{2} + 0. 5^{2}) = 0.5$ . Child L: $p_{0} = 7/8, p_{1} = 1/8$ → $i_{L} = 1 - (0.7656 + 0.01563) = 0.21875$ . Child R: $p_{0} = 1/8, p_{1} = 7/8$ → $i_{R} = 1 - (0.01563 + 0.7656) = 0.21875$ . Weighted children: $\frac{8}{16} (0.21875) + \frac{8}{16} (0.21875) = 0.21875$ . $Δ i (root) = 0.5 - 0.21875 = 0.28125$ . $Imp (F_{1}) = 1 \times 0.28125 = 0.28125$ .

Node-L split (on $F_{2}$ ), arrival weight $= 8/16 = 0.5$ . $i (L) = 0.21875$ (from above). Leaf LL: pure → $i_{LL} = 0$ . Leaf LR: $p_{0} = 3/4, p_{1} = 1/4$ → $i_{L R} = 1 - (0.5625 + 0.0625) = 0.375$ . Weighted children: $\frac{4}{8} (0) + \frac{4}{8} (0.375) = 0.1875$ . $Δ i (L) = 0.21875 - 0.1875 = 0.03125$ . $Imp (F_{2}) = 0.5 \times 0.03125 = 0.015625$ .

Normalise. Total $= 0.28125 + 0.015625 = 0.296875$ . $Imp (F_{1}) = \frac{0.28125}{0.296875} = 0.9474, Imp (F_{2}) = \frac{0.015625}{0.296875} = 0.0526.$ $F_{1}$ does $\approx 95%$ of the impurity work — it separated the two classes at the root; $F_{2}$ only mopped up a nearly-pure node reached by half the data. This is the same Decision Trees machinery, just totalled per feature.

L4.2 — regression synthesis

A regression tree: root holds $y = {2, 2, 8, 8}$ ( $N = 4$ ), splits on $F_{3}$ into ${2, 2}$ and ${8, 8}$ . Compute $Imp (F_{3})$ (root split, weight $= 1$ ).

Recall Solution

Root: $\overset{y}{ˉ} = 5$ , deviations $^{2} = 9, 9, 9, 9$ , sum $36$ , $i = \frac{36}{4} = 9$ . Each child is constant → variance $0$ . Weighted children $= \frac{1}{2} (0) + \frac{1}{2} (0) = 0$ . $Δ i = 9 - 0 = 9$ . Arrival weight $1$ → $Imp (F_{3}) = 9$ . A perfect split (each child constant) removes all variance — the maximum possible cleanup.

Level 5 — Mastery

L5.1

You fit a Random Forest on customer data. A customer_id column (a unique integer per row, thousands of distinct values) shows the highest MDI. Is customer_id your most predictive feature? Diagnose and prescribe.

Recall Solution

No. customer_id has enormous cardinality, so it offers a split point between almost every pair of rows. With so many candidate splits, it can carve the training data into near-pure leaves by memorising, driving up impurity decrease on the training set even though it carries no generalisable signal — a classic MDI cardinality bias compounded by being computed on training data (overfitting; see Bias-Variance Tradeoff). Prescribe: (1) drop identifier columns before training; (2) rank features with Permutation Importance on a held-out set (shuffle the column, measure the score drop — an id column shuffled changes nothing, so its permutation importance collapses to ~0); (3) cross-check with SHAP Values.

L5.2

Your MDI ranking says feature $D$ is worthless (importance $\approx 0$ ), but a domain expert insists $D$ is highly predictive. Under what single condition can both be right, and how do you confirm it?

Recall Solution

Both are right if $D$ is strongly correlated with another feature $C$ that the trees happened to use instead. Once $C$ is split on, $D$ becomes redundant and never gets selected, so its MDI is ~0 — yet $D$ alone would predict well. Confirm: (i) compute the correlation / clustering of $D$ with used features; (ii) run grouped permutation importance (permute the ${C, D}$ cluster together) — if the cluster's importance is high, the signal lives in ${C, D}$ jointly; (iii) re-fit with $C$ removed and watch $D$ 's MDI jump. This is exactly the correlated-feature failure mode from the parent note's Mistake 2.

L5.3

Choose the right tool. For each goal below, pick MDI, permutation importance, or SHAP, and justify in one line. (a) A fast, free ranking during model development. (b) An honest ranking for a report to stakeholders. (c) Per-prediction, signed attribution ("this loan was denied because income was low").

Recall Solution

(a) MDI — it's computed for free while the tree grows; fine for quick internal iteration, just distrust high-cardinality winners. (b) Permutation importance — model-agnostic, computed on held-out data, so it doesn't reward training overfit; the fair choice for a stakeholder ranking. (c) SHAP — the only one giving signed, per-sample contributions with additive game-theoretic guarantees, so it can say why this individual prediction went the way it did.

Connections

Feature importance from trees — the parent note these exercises drill.
Decision Trees — every $Δ i$ here is a split the tree already made.
Gini vs Entropy — swap $i (t)$ and the numbers change, the method doesn't.
Random Forests — L5.1 lives here (average over trees).
Gradient Boosting — gain-based importance shares MDI's biases.
Permutation Importance — the fix in L5.
SHAP Values — per-prediction attribution in L5.3.
Bias-Variance Tradeoff — why training-set MDI overestimates.