Feature importance from trees — Hinglish notes

2.3.8 · AI-ML › Tree-Based & Instance Methods

KYA compute kar rahe hain hum?

Teen ingredients:

Impurity $i (t)$ : ek node kitna mixed hai (classification ke liye Gini ya entropy, regression ke liye variance/MSE).
Impurity decrease $Δ i (t)$ : split ne node ko kitna saaf kiya.
Weighting $N_{t} / N$ : tree mein upar ka split zyada samples affect karta hai → zyada count hota hai.

KAISE: $Δ i (t)$ first principles se derive karo

WHY children ko weight karte hain? Ek parent node jisme $N_{t}$ samples hain woh ek left child ( $N_{t_{L}}$ ) aur right child ( $N_{t_{R}}$ ) mein split hota hai. Hum parent ki impurity ko children ki average impurity se compare karte hain — lekin jis child mein zyada samples hain woh zyada count karna chahiye. Isliye hum children ko unke sample fractions se weight karte hain:

$Δ i (t) = i (t) - \frac{N _{t_{L}}}{N _{t}} i (t_{L}) - \frac{N _{t_{R}}}{N _{t}} i (t_{R})$

Yeh hamesha $\geq 0$ kyun hai: Greedy tree algorithms woh split choose karte hain jo is drop ko maximize kare, aur impurity ek concave function hai, isliye splitting kabhi (weighted) impurity badha nahi sakti. Isliye importances non-negative hoti hain.

Ab arrival weight $N_{t} / N$ se multiply karo (fraction of all data jo node $t$ tak pahunchta hai) aur $f$ par split karne wale har node pe sum karo → is se upar wala $Imp (f)$ milta hai.

Forests: trees par average karo

Ek Random Forest / GBM jisme $T$ trees hain, importance trees ka mean hoti hai, usually 1 sum karne ke liye normalize ki gayi: $Imp_{forest} (f) = \frac{1}{T} \sum_{τ = 1}^{T} Imp_{τ} (f), Imp (f) = \frac{Imp ( f )}{\sum _{g} Imp ( g )}$

Worked Example 1 — ek split, Gini

Root mein 10 samples hain: 5 positive, 5 negative. Feature $X_{1}$ par split se milta hai:

Left: 4 samples (4 pos, 0 neg)
Right: 6 samples (1 pos, 5 neg)

Step 1 — parent impurity. Kyun? Baseline mixed-ness. $i (root) = 1 - (0. 5^{2} + 0. 5^{2}) = 0.5$

Step 2 — child impurities. Kyun? Measure karo ki har side kitni clean hui. $i (L) = 1 - (1^{2} + 0^{2}) = 0$ $i (R) = 1 - ((1/6)^{2} + (5/6)^{2}) = 1 - (0.0278 + 0.6944) = 0.2778$

Step 3 — weighted child impurity. Kyun? Bada child zyada count karta hai. $\frac{4}{10} (0) + \frac{6}{10} (0.2778) = 0.1667$

Step 4 — impurity decrease. Kyun? Jo cleanup achieve hua. $Δ i = 0.5 - 0.1667 = 0.3333$

Step 5 — arrival weight. Root sabhi data dekhta hai, $N_{t} / N = 10/10 = 1$ . $Imp (X_{1}) = 1 \times 0.3333 = 0.3333$

Worked Example 2 — deeper node weighting

Maan lo $X_{2}$ ek aisi node par split karta hai jo sirf 10 mein se 3 samples tak pahunchi hai, aur $Δ i = 0.4$ hai.

Step — arrival weight lagao. Kyun? Jo split thode samples affect kare woh dominate nahi karni chahiye. $Imp (X_{2}) = \frac{3}{10} \times 0.4 = 0.12$

Zyada local $Δ i$ hone ke bawajood (0.4 > 0.3333), $X_{2}$ (0.12) ka rank $X_{1}$ (0.3333) se neeche hai kyunki $X_{1}$ ki split ne sabko touch kiya. Yahi reason hai $N_{t} / N$ ka.

Normalized: $Imp (X_{1}) = \frac{0.3333}{0.4533} = 0.735$ , $Imp (X_{2}) = 0.265$ .

Worked Example 3 — regression (variance drop)

Node jisme $y = {2, 4, 6, 8}$ , $\overset{y}{ˉ} = 5$ hai. Split into ${2, 4}$ aur ${6, 8}$ .

Step 1 — parent variance. $\frac{( 9 + 1 + 1 + 9 )}{4} = 5$ Step 2 — child variances. Har child: $\overset{y}{ˉ} = 3$ ya $7$ , var $= \frac{1 + 1}{2} = 1$ . Step 3 — weighted. $\frac{1}{2} (1) + \frac{1}{2} (1) = 1$ Step 4 — decrease. $Δ i = 5 - 1 = 4$ . Bada drop → yeh feature $y$ ko achhi tarah separate karta hai.

Common mistake Galat ideas ko steel-man karna

Mistake 1: "High MDI = truly predictive feature." Kyun sahi lagta hai: feature ne bahut impurity ka kaam kiya. The trap: MDI high-cardinality / continuous features ki taraf biased hai — unke paas zyada split points hote hain, isliye woh noise fit karke bhi impurity reduce kar sakte hain. Fix: permutation importance use karo (feature ko held-out data par shuffle karo, score drop measure karo) ya kam biased view ke liye SHAP.

Mistake 2: "Correlated features importance fairly share karte hain." Kyun sahi lagta hai: dono signal carry karte hain. The trap: ek baar ek use ho jaye, toh uska correlated twin redundant ho jaata hai aur near-zero MDI paata hai, ek real predictor chhupa leta hai. Fix: correlated features ko cluster karo ya permutation importance grouped by cluster use karo.

Mistake 3: $N_{t} / N$ weight bhool jaana. Tab deep, tiny-sample splits utne hi important lagte hain jitna root. Hamesha node tak pahunchne wale samples se weight karo.

Mistake 4: MDI training data par compute ki gayi importance overestimate karta hai (overfitting). Honest ranking ke liye held-out permutation importance prefer karo.

Recall Feynman: 12-saal ke bachche ko samjhao

Socho tum ek messy box mein red aur blue marbles ko saaf piles mein sort kar rahe ho. Har "sawaal" jo tum poochho (jaise kya yeh ek coin se bada hai?) jo piles ko neater banaye, woh points earn karta hai. Ek sawaal jo SAARE marbles ko perfect red-only aur blue-only piles mein split kare woh bahut points earn karta hai; ek sawaal jo sirf 3 marbles ki help kare woh kam points paata hai. Puri game mein har sawaal-type ke points add karo — sabse zyada points wale sawaal most important features hain. Lekin savdhan raho: ek sawaal jiske bahut saare possible answers hain (jaise grams mein exact weight) "cheat" kar sakta hai aur important lag sakta hai jab woh sirf memorize kar raha ho.

Flashcards

MDI (Mean Decrease in Impurity) kya measure karta hai?

Ek feature ka use karne wale sabhi splits se total weighted impurity reduction, tree(s) par sum ki gayi.

Node par weighted impurity decrease likhो.

Δ i (t) = i (t) - \frac{N _{t_{L}}}{N _{t}} i (t_{L}) - \frac{N _{t_{R}}}{N _{t}} i (t_{R})

Har node ke

Δ i

N_{t} / N

se multiply kyun karte hain?

Taaki zyada samples tak pahunchne wale splits (usually tree mein upar) thode samples affect karne wale deep splits se zyada count karein.

Gini impurity formula do aur kyun kaam karta hai.

1 - \sum_{k} p_{k}^{2}

; yeh woh probability hai ki node se do random draws class mein disagree karein — mixedness ka ek measure.

Impurity decrease hamesha non-negative kyun hoti hai?

Greedy algorithm woh split choose karta hai jo (concave) impurity drop maximize kare; splitting weighted impurity badha nahi sakti.

Regression trees mein kaun si impurity use hoti hai?

Node mean ke baare mein Variance / MSE:

\frac{1}{N _{t}} \sum (y - \overset{y}{ˉ}_{t})^{2}

MDI feature importance ka main bias kya hai?

High-cardinality/continuous features ke liye inflated (zyada split points → noise fit kar sakta hai); training data par compute hone par bhi biased.

MDI ke biases ka fix kya hai?

Permutation importance use karo (feature ko held-out data par shuffle karo) ya SHAP values.

Do correlated features ki importance mein kya hota hai?

Ek baar ek use ho jaye, doosra redundant ho jaata hai aur near-zero MDI paata hai, ek real predictor chhupa leta hai.

Trees ki importances se forest importance kaise milti hai?

Sabhi trees par per-tree importances ka average lo, phir 1 sum karne ke liye normalize karo.

Connections

Decision Trees — MDI usi impurity ka byproduct hai jo tree grow karne mein use hoti hai.
Gini vs Entropy — $i (t)$ ka choice importance values badalta hai.
Random Forests — stability ke liye bahut saare trees par importance average ki jaati hai.
Gradient Boosting — split-based importances bhi expose karta hai (gain/cover/frequency).
Permutation Importance — unbiased, model-agnostic alternative.
SHAP Values — MDI biases fix karne wala game-theoretic attribution.
Bias-Variance Tradeoff — training MDI overfitting ki wajah se overestimate karta hai.