Feature importance from trees: Question bank — AI-ML deep dive

Before the traps, we pin down the exact formula and vocabulary the whole bank leans on — so every "spot the error" is checked against something written, not from memory.

The figure below shows one split so all these symbols have a picture attached before you meet them in the traps.

And here is why the impurity drop can never be negative — the concavity / Jensen picture, made visual instead of a one-liner.

True or false — justify

MDI importance can be negative for a feature that hurt the model.

False — the greedy algorithm only accepts the split that maximises the (concave) impurity drop, so every accepted

Δ i (t) \geq 0

; summing non-negatives can never go below zero.

A feature with importance exactly 0 was never chosen for any split in the tree.

True — importance is a sum over nodes that split on that feature; if the feature split zero nodes the sum is empty, hence 0. (It does not mean the feature is useless — a correlated twin may have been chosen instead.)

If you rebuild the same tree with entropy instead of Gini, the ranking of features is guaranteed identical.

False — different impurity functions

i (t)

give different

Δ i

values (see Gini vs Entropy), so both the numbers and occasionally the ordering can shift, especially for close calls.

Normalised MDI importances always sum to 1 across all features.

True by construction — we divide each raw importance by

\sum_{g} Imp (g)

; this is a convenience so importances read like percentages, not a property of the data.

Doubling every sample's copy (duplicating the whole dataset) leaves normalised MDI unchanged.

True — every

N_{t}

and

N

doubles, so each ratio

N_{t} / N

and each proportion

p_{k}

is identical; the impurity landscape is scale-invariant to uniform replication.

A high-importance feature must generalise well to new data.

False — MDI is measured on the training fit. A noisy high-cardinality feature can rack up impurity drops by memorising, so high MDI ≠ predictive on held-out data (that's why Permutation Importance exists).

For a forest, a feature's importance is the sum of its per-tree importances.

False — it's the mean over trees (then usually normalised); summing would just scale with the number of trees

T

and hide nothing extra. See Random Forests.

Removing the

N_{t} / N

weight and only summing

Δ i (t)

still gives a fair importance.

False — then a deep split touching 3 samples counts the same as the root touching everyone; the arrival weight is exactly what discounts low-impact deep splits.

Spot the error

"Feature

X

has the biggest

Δ i

at its node, so it is the most important feature."

The error is ignoring the arrival weight. In the s01 figure, the drop

Δ i

is a local quantity at one box; importance multiplies it by

N_{t} / N

, the share of all marbles that reached that box. A huge

Δ i

at a deep node reaching 3 samples can rank below a modest

Δ i

at the root reaching all 10.

"Two perfectly correlated features carry the same signal, so MDI splits the credit 50/50 between them."

Wrong — the tree greedily picks one; once used, its twin becomes redundant and offers little further impurity drop (there's little impurity left on that path to remove), so the twin often gets near-zero MDI. Credit is grabbed by whoever was chosen first, not shared.

"MDI is model-agnostic — I can compute it for any classifier."

MDI is defined only for tree-based models because the sum in the boxed formula runs over concrete splits with impurity drops. For non-tree models (SVM, neural nets) there are no such splits, so you need Permutation Importance or SHAP Values.

"I computed MDI on the training set, so my importance ranking is unbiased."

Training MDI overestimates importance because the impurity drops in the s02-style hill picture were achieved partly by fitting noise — an overfitting effect (see Bias-Variance Tradeoff). An honest ranking uses held-out permutation importance.

"A continuous feature and a binary feature with the same true signal will get equal MDI."

No — the continuous feature offers many candidate thresholds, so it gets more chances to find some chord below the impurity hill (even from noise). MDI is biased toward high-cardinality/continuous features.

Δ i (t) = i (t) - i (t_{L}) - i (t_{R})

Missing the child weights

N_{t_{L}} / N_{t}

and

N_{t_{R}} / N_{t}

. In the concavity picture,

Δ i

is the gap between the parent point and the weighted-average of the children on the chord. Drop the weights and you're subtracting two full impurities — you could even get a nonsensical negative "drop".

Why questions

Why do we weight the children by

N_{t_{L}} / N_{t}

and

N_{t_{R}} / N_{t}

inside

Δ i

Because the children partition the parent's

N_{t}

samples; each child should count in proportion to how many samples it holds. Geometrically (s02),

Δ i

is the gap to the weighted chord point — an unweighted average would sit at the wrong place and a tiny pure child would falsely make any split look great.

Why is impurity decrease guaranteed non-negative even before choosing the best split?

Impurity is a concave (hill-shaped) function of the class distribution — see the s02 figure. The children's weighted-average impurity lands on the chord joining their two points, which lies below the hill where the parent sits. So the drop = parent minus chord

\geq 0

; this is Jensen's inequality made visual.

Why does a split high in the tree tend to matter more than a deep one?

A high split has a large arrival weight

N_{t} / N

(nearly all samples reach it), so its impurity drop is amplified in the boxed sum. Deep splits reach few samples, so even a big local

Δ i

is scaled down.

Why does the Gini form

\sum_{k} p_{k} (1 - p_{k})

measure "mixedness"?

p_{k} (1 - p_{k})

is the chance of drawing class

k

then a non-

k

item; summing gives the probability two random draws from the node disagree. A pure node has zero disagreement, hence zero impurity — the ends of the hill in s02.

Why use variance instead of Gini for regression trees?

Regression targets are continuous, not classes, so "class proportions" don't exist. The natural impurity is the leftover squared error about the node mean

\overset{y}{ˉ}_{t}

— the variance the split has yet to explain.

Why does averaging importances over many trees in a forest give a more trustworthy ranking?

Single trees are high-variance; which feature wins a near-tie split is noisy. Averaging over de-correlated trees smooths out that noise, giving stable estimates (see Random Forests).

Why can Gradient Boosting report several different importances (gain, cover, frequency)?

"Gain" is the MDI-style impurity/loss reduction, "cover" weights by samples touched, and "frequency" just counts how often a feature is used. They answer different questions, so they can rank features differently.

Edge cases

A node is already pure (

i (t) = 0

) but the algorithm still splits it. What importance does that split add?

Zero — you can't decrease impurity below zero, so

Δ i = 0

and the contribution

\frac{N _{t}}{N} \cdot 0 = 0

. Pure nodes sit at the bottom corners of the s02 hill and are normally not split at all.

A feature is used in two different nodes of the same tree. How is its importance formed?

You sum the weighted drops from both nodes:

\sum_{t : split (t) = f} \frac{N _{t}}{N} Δ i (t)

. A feature reused at several places accumulates importance across all of them.

Every feature is pure noise and the tree is depth-limited to a single root that can't beat the no-split baseline. What are the importances?

If no split reduces impurity, no split is made, all sums are empty, and every feature's importance is 0 (normalisation of all-zeros is undefined, typically reported as 0).

A dataset has one feature that perfectly separates the classes at the root. What do the other features' MDIs look like?

The root split already drives impurity to (near) zero, leaving almost no impurity for lower splits to remove — so other features get tiny or zero MDI even if they carry real, redundant signal.

Two classes are perfectly balanced (50/50) in a binary problem — what is the maximum possible Gini at the root?

1 - (0. 5^{2} + 0. 5^{2}) = 0.5

, the largest Gini for two classes — the very top of the s02 hill. Maximum mixedness means the split has the most room to reduce impurity.

You duplicate one feature column and refit. What happens to that feature's MDI compared with before?

It roughly halves per copy — the tree can pick either identical column at each split, so the impurity credit is randomly divided between the two twins, understating the feature's true role.

Connections

Feature importance from trees — the parent note this bank drills.
Decision Trees — where the impurity drops come from.
Gini vs Entropy — why the choice of $i (t)$ shifts values.
Random Forests — averaging importances over trees.
Gradient Boosting — gain / cover / frequency importances.
Permutation Importance — the unbiased, held-out alternative.
SHAP Values — game-theoretic fix for MDI biases.
Bias-Variance Tradeoff — why training MDI overestimates.