Feature importance from trees: Foundations — AI-ML deep dive

This page builds every symbol the parent note Feature importance from trees throws at you, starting from absolute zero. If the parent wrote $\frac{N _{t}}{N} Δ i (t)$ and you froze — this page unfreezes you, one symbol at a time.

0. The picture behind everything: a tree sorting marbles

Before any symbol, hold this picture in your head.

Look at the figure. At the top is a box holding a mix of coral and mint marbles — messy. We ask a question ("is the marble bigger than a coin?"), and it splits the box into two smaller boxes below. Each smaller box is (hopefully) less mixed — more of one colour. Every circle here is called a node. The top node is the root, the boxes below are its children.

That's the whole game. Everything else is just notation for "how messy", "how much less messy", and "how much data was affected".

1. The data: samples, $N$ , and features

Why the topic needs these. Later we weight a question by "how many marbles it touched", which is exactly $N_{t}$ out of $N$ . So $N$ and $N_{t}$ are the raw counts that make weighting possible.

2. The subscript language: $t$ , $t_{L}$ , $t_{R}$

The little letters attached to $N$ and $i$ are just labels for which box we mean.

Read the figure: one parent box $t$ with $N_{t}$ marbles pours into a left box $t_{L}$ (holding $N_{t_{L}}$ ) and a right box $t_{R}$ (holding $N_{t_{R}}$ ). Every marble goes to exactly one side, so the counts always add up:

$N_{t_{L}} + N_{t_{R}} = N_{t}$

3. Proportions: $p_{k}$ and $\sum$

The parent note writes $\sum_{k} p_{k}^{2}$ . Two new symbols: $p_{k}$ and $\sum$ .

Why the topic needs this. "Messiness" (impurity) will be built entirely out of these proportions — a box that is all one colour has one $p_{k} = 1$ and the rest $0$ ; a box that is evenly split has small $p_{k}$ everywhere. The sum-of-squares captures that difference.

4. Impurity $i (t)$ — putting a number on "messy"

Now we can define the star symbol: $i (t)$ , read "the impurity of node $t$ ."

The figure shows three boxes and their Gini impurity $1 - \sum_{k} p_{k}^{2}$ :

All coral → $i = 1 - 1^{2} = 0$ (pure, floor value).
3 coral / 1 mint → $i = 1 - 0.625 = 0.375$ .
Even 2 / 2 → $i = 1 - (0. 5^{2} + 0. 5^{2}) = 0.5$ (the maximum mess for two colours).

For number columns (regression) the messiness meter is variance — how spread out the numbers are around their average. Same idea, different flavour; the parent uses $i_{MSE}$ there.

5. The average bar: $\overset{y}{ˉ}_{t}$

The regression formula uses $\overset{y}{ˉ}_{t}$ .

6. Impurity decrease $Δ i (t)$ — the payoff of a question

This is the heart of the topic. New symbol: $Δ$ (Greek capital delta).

WHAT we just did: subtracted the children's messiness from the parent's messiness. WHY the fractions $\frac{N _{t_{L}}}{N _{t}}$ : they are the shares of marbles going left and right. A child holding more marbles should count more toward the "after" mess — so we weight each child's impurity by its share. These two fractions add to $1$ , so the bracket is a genuine weighted average. WHAT IT LOOKS LIKE: the tall parent bar shrinking down to two shorter child bars in the figure below.

In the figure, the coral bar (parent impurity $0.5$ ) is the "before". The two mint bars are the children; their sample-weighted average is the "after" line. The green gap between them is $Δ i (t)$ — the cleanup earned by this one question.

7. The arrival weight $\frac{N _{t}}{N}$

We now know a question's local payoff $Δ i (t)$ . But a question near the root touches everyone; a question deep down touches a handful. To be fair we scale by how much of all the data reached that node.

8. Assembling the parent's formula

Now every piece of $Imp (f) = \sum_{t : split (t) = f} \frac{N _{t}}{N} Δ i (t)$ is de-mystified:

Symbol	Plain words	Picture
$Imp (f)$	importance of feature $f$	total points feature $f$ scored
$\sum_{t : split (t) = f}$	add over every node that asked about $f$	walk the tree, collect $f$ 's boxes
$\frac{N _{t}}{N}$	slice of all data reaching that box	pie fraction
$Δ i (t)$	cleanup at that box	shrinking-bar gap

For a whole forest you just average $Imp (f)$ over all trees and normalize so the importances sum to $1$ . That normalization, biases, and better alternatives (Permutation Importance, SHAP Values) are the parent note's job — you now have the vocabulary to read it.

9. Prerequisite map

Read it bottom-up: counts and proportions build impurity; impurity builds decrease; decrease plus weight builds importance.

Equipment checklist

Cover the right side and answer before revealing.

What is

N

versus

N_{t}

N

= total samples in the whole dataset (fixed);

N_{t}

= samples that reach node

t

(shrinks with depth).

What do

t

t_{L}

t_{R}

label?

A node (parent) and its left and right child boxes — subscripts are name tags, not multiplication.

What is

p_{k}

and what range does it live in?

The fraction of samples in a box that are class

k

; between 0 and 1, and all

p_{k}

in a box sum to 1.

What does

\sum_{k} p_{k}^{2}

compute?

Add up the square of each class proportion — the probability two random draws match colour.

What does

i (t) = 0

mean?

The box is perfectly pure — all one class, nothing left to clean.

What does the symbol

Δ

mean in

Δ i (t)

"Change/drop" — impurity before the split minus the weighted impurity after.

Why are children weighted by

N_{t_{L}} / N_{t}

and

N_{t_{R}} / N_{t}

Bigger children hold more samples and should count more; the two fractions form a weighted average that sums to 1.

Why is

Δ i (t) \geq 0

The greedy tree only accepts splits that reduce impurity, and impurity is concave, so splitting can't increase weighted messiness.

What is the arrival weight

N_{t} / N

and its value at the root?

The fraction of the whole dataset reaching node

t

; at the root it equals 1.

How do you get forest importance from tree importances?

Average the per-tree importances across all trees, then normalize to sum to 1.