Gini impurity

2.3.3 · AI-ML › Tree-Based & Instance Methods

Ek decision tree data ko split karta hai taaki har resulting group jitna ho sake utna "pure" ho — ideally har leaf mein sirf ek hi class ho. Gini impurity ek number hai jo measure karta hai ki ek node kitna impure (mixed-up) hai. Kam = zyada pure.

The Core Idea

YE FORMULA KYU? (Scratch se Derivation)

HUM KYA CHAHTE HAIN: woh probability ki ek randomly picked item ko galat label assign ho jab hum node ki apni distribution se ek random class label draw karke assign karte hain.

HUM ISSE STEP BY STEP KAISE BANATE HAIN:

Ek random item uthao. Probability ki woh sach mein class $i$ se belong karta hai woh $== p_{i} ==$ hai. Kyun? $p_{i}$ usi class ka node mein share hai, yahi uski definition hai.
Usi distribution se ek random label independently uthao. Us label ke class $i$ hone ki probability bhi $p_{i}$ hai.
Hum galat hote hain jab bhi true class ( $i$ ) aur guessed label ( $j$ ) alag hon: $i \neq = j$ . Class $i$ par match hone ki probability $p_{i} \cdot p_{i} = p_{i}^{2}$ hai. Square kyun? Do independent draws dono $i$ par land karte hain.
Kisi bhi class par correct match hone ki total probability: $\sum_{i} p_{i}^{2}$ .
Isliye galti hone ki probability: $G = 1 - \sum_{i = 1}^{K} p_{i}^{2}$
Algebraically rewrite karo taaki "per-class error" view dikhe: $1 - \sum_{i} p_{i}^{2} = \sum_{i} p_{i} - \sum_{i} p_{i}^{2} = \sum_{i} p_{i} (1 - p_{i})$ Yeh kyun matter karta hai? Har term $p_{i} (1 - p_{i})$ = (item ke class $i$ hone ki chance) × (hamara guess class $i$ NOT hone ki chance). Yeh padhta hai "har class ke liye expected misclassification."

Common Mistakes

Recall Feynman: explain to a 12-year-old

Tumhare paas jelly beans ka ek box hai, kuch red, kuch green. Tum ek game khelte ho: ek bean nikalo, uska color chhupao, phir doosri random bean nikal ke uska color copy karke guess karo. Agar box mein saari red hain, tum hamesha jeet te ho. Agar aadhi-aadhi hain, tum bahut baar haarte ho. Gini impurity bas "kitni baar tum yeh game haarte ho" hai. Ek smart tree box ko chhote-chhote boxes mein split karta rehta hai taaki har box mein almost sirf ek hi color ho — kyunki phir tum almost kabhi nahi haarte.

Active Recall

Ek node ke liye Gini impurity formula kya hai?

G = 1 - \sum_{i = 1}^{K} p_{i}^{2} = \sum_{i} p_{i} (1 - p_{i})

Gini impurity literally kya measure karta hai (probabilistic meaning)?

Ek random item ko misclassify karne ki probability, agar tum uski class node ki apni class distribution se randomly drawn label use karke guess karo.

Perfectly pure node ki Gini impurity kya hoti hai?

K classes ke liye maximum Gini impurity kya hai, aur kab?

1 - 1/ K

, jab saari classes equally likely hon (uniform

p_{i} = 1/ K

2-class node ke liye Max Gini?

0.5 (50/50 split par).

Hum ek candidate split ko kaise score karte hain?

Children ki weighted average impurity

\sum_{c} \frac{N _{c}}{N} G_{c}

; tree Gini gain maximize karta hai =

G_{p a r e n t} - G_{s pl i t}

Children ko plain average ki jagah

N_{c} / N

se weight kyun karte hain?

Bade children zyada total misclassifications contribute karte hain; unweighted sums child sizes ko ignore karte hain aur zyada branches ke saath inflate hote hain.

Uniform distribution worst (highest Gini) kyun hai?

Kyunki

\sum p_{i}^{2}

tab minimize hota hai jab probability evenly spread ho, isliye

G = 1 - \sum p_{i}^{2}

maximize hota hai.

Gini vs Entropy — practical difference?

Dono purity par 0 hain, uniform par max; Gini

lo g

compute karne se bachta hai isliye faster hai aur usually near-identical trees deta hai.

Kya training data par Gini=0 ka matlab achha model hai?

Nahi — yeh often overfitting/memorization signal karta hai; purity split criterion hai, generalization target nahi.

Connections

Decision Trees — Gini CART/scikit-learn mein default split criterion hai.
Entropy and Information Gain — alternative impurity measure ( $- \sum p lo g p$ ).
Overfitting and Pruning — kyun hum Gini=0 chase nahi karte.
Random Forests — bahut saare Gini-grown trees ka average.
Classification vs Regression Trees — regression Gini ki jagah variance/MSE use karta hai.
Gini Coefficient (Economics) — alag concept, same naam (confuse mat karo).