Gini impurity: Question bank — AI-ML deep dive

Every symbol below is from the parent: $p_{i}$ = the proportion (share) of class $i$ in a node, $K$ = number of classes, $G$ = Gini impurity $= 1 - \sum_{i = 1}^{K} p_{i}^{2}$ , $N_{c} / N$ = fraction of parent samples landing in child $c$ .

Two pictures anchor everything below — keep them in view:

The parabola above is $G = 2 p (1 - p)$ for a two-class node, where $p$ is one class's share. It is $0$ at the pure ends ( $p = 0$ and $p = 1$ ) and peaks at $G = 0.5$ dead center ( $p = 0.5$ ). This single curve explains EC6, TF3 (mirror symmetry about $p = 0.5$ ), and WHY5 (the curve bulges upward, i.e. it is concave — a chord between two points sits below it).

The right panel warns about SE5: two different "Gini" ideas. ML's Gini impurity is the parabola on the left; the economics Gini coefficient is an area ratio on a Lorenz curve. Same name, unrelated construction.

True or false — justify

TF1. "A node with

G = 0

is always a leaf where the tree stops."

False.

G = 0

means the node is pure so there's nothing left to gain, so the tree won't split it — but stopping can also happen earlier (depth limit, min-samples). Purity forces a stop; it isn't the only reason to stop.

TF2. "Adding a class that has proportion

p_{i} = 0

changes the Gini value."

False. A zero-proportion class contributes

p_{i}^{2} = 0

to the sum, so

G

is unchanged. Only classes that actually appear affect impurity.

TF3. "For two classes,

G

is symmetric: an 80/20 node and a 20/80 node have the same impurity."

True.

G = 1 - p^{2} - (1 - p)^{2}

depends only on the set of proportions, not which class is which — the parabola in figure s01 is mirror-symmetric about

p = 0.5

. Swapping labels leaves

\sum p_{i}^{2}

identical.

TF4. "Gini impurity can be greater than 1."

False. Its maximum is

1 - \frac{1}{K}

, which stays strictly below 1 for every finite

K

and only approaches 1 as

K \to \infty

. Since

\sum p_{i}^{2} \geq \frac{1}{K} > 0

G

is always strictly below 1.

TF5. "If a split makes both children more impure than the parent, the tree will still take it because splitting is always progress."

False. If the weighted child impurity

G_{split}

exceeds the parent's, Gini gain is negative and the greedy tree rejects that split. A split must reduce expected error to be chosen.

TF6. "Two different splits with the same

G_{split}

are equally good to the tree."

True by the Gini criterion alone — the tree ranks purely on gain

G_{parent} - G_{split}

, so equal

G_{split}

ties. (A tie-breaking rule, like feature order, then decides, but Gini itself is indifferent.)

TF7. "Gini impurity depends on how many samples

N

are in the node."

False.

G

uses only the proportions

p_{i}

, not counts. A node of 3 samples and a node of 300 with the same class ratios have identical

G

. Counts matter only in the weighted split score

\frac{N _{c}}{N} G_{c}

TF8. "The maximum Gini for 4 classes is larger than for 2 classes."

True. Max is

1 - \frac{1}{K}

: for

K = 2

it's

0.5

, for

K = 4

it's

0.75

. More classes means more ways to be wrong, so peak impurity rises.

Spot the error

SE1. "This split has children with Gini 0.1 and 0.6, so the split's impurity is

0.1 + 0.6 = 0.7

Error: children are averaged with weights

\frac{N _{c}}{N}

(the

G_{split}

formula above), never summed. Raw sums have no probabilistic meaning and would exceed 1 with enough branches.

SE2. "Node has 3 apples and 1 orange, so

G = 1 - (3^{2} + 1^{2}) = 1 - 10 = - 9

Error: you must square proportions, not raw counts. Correct:

p = (\frac{3}{4}, \frac{1}{4})

G = 1 - \frac{9}{16} - \frac{1}{16} = \frac{6}{16} = 0.375

. A negative Gini is an impossible red flag.

SE3. "Gini gain

= G_{split} - G_{parent}

, and we pick the split with the largest gain."

Error: the subtraction is backwards. Gain

= G_{parent} - G_{split}

; splitting lowers impurity, so parent minus child is the positive improvement.

SE4. "For a 50/50 two-class node,

G = 1 - (0.5 + 0.5)^{2} = 1 - 1 = 0

Error: you squared the sum instead of summing the squares. Correct:

1 - (0. 5^{2} + 0. 5^{2}) = 1 - 0.5 = 0.5

— the maximum (the parabola's peak in s01), not zero.

SE5. "This is the ML Gini, so I'll draw the Lorenz curve to compute it."

Error: the Lorenz curve belongs to the economics Gini coefficient (right panel of s02, see Gini Coefficient (Economics)). ML's Gini impurity is just

1 - \sum p_{i}^{2}

, a misclassification probability with no Lorenz curve involved.

SE6. "The parent has

G = 0.5

; a child with only 1 sample is pure so

G_{c} = 0

, which alone proves this is the best split."

Error: a single-sample pure child gets tiny weight

\frac{1}{N}

, so it barely lowers

G_{split}

. Purity of a large child is what drags the weighted average down. Size matters.

SE7. "Entropy uses

lo g

and Gini doesn't, so they produce completely different trees."

Error: both are 0 at purity and maxed at uniform, and agree on splits roughly 98% of the time. Gini skips

lo g

mainly for speed, not to get different trees.

Why questions

WHY1. "Why do we square the proportions instead of just using

\sum p_{i}

Because

\sum p_{i} = 1

always — it carries no information. Squaring rewards concentration: a dominant class contributes a large

p_{i}^{2}

, shrinking

G

, which is exactly the "how often do I win the guessing game" signal we want.

WHY2. "Why is a uniform distribution the worst (highest Gini), not the best?"

\sum p_{i}^{2}

is minimized when probability is spread evenly, so

G = 1 - \sum p_{i}^{2}

is maximized there. Even spread means no class to reliably guess, so the random-guess mistake rate peaks.

WHY3. "Why weight children by

N_{c} / N

rather than treating each child equally?"

Because we care about total expected misclassifications across all data. A big impure child produces many mistakes; a tiny pure one produces few. Equal weighting would let a small child overrule a large one.

WHY4. "Why doesn't the tree just keep splitting until every leaf has

G = 0

Driving training Gini to 0 usually means memorizing noise — overfitting (see Overfitting and Pruning). Purity is the split criterion, not the generalization goal; that's why we prune and cap depth.

WHY5. "Why can Gini gain never be negative for the chosen split? (What does 'concave' mean here?)"

'Concave' means the graph

G (p) = 1 - \sum p_{i}^{2}

bulges upward, so any straight chord between two points lies below the curve — visible in the parabola s01. A split replaces the parent's point on the curve with a weighted average (a point on the chord) of the children's points; by concavity (Jensen's inequality) that chord value is

\leq

the value at the averaged

p

, which is the parent. So

G_{split} \leq G_{parent}

, and gain

\geq 0

for the best split.

WHY6. "Why is Gini preferred over Entropy in Random Forests and CART by default?"

It avoids computing logarithms, so it's cheaper per split — and across thousands of trees that speed adds up while giving near-identical results (see Entropy and Information Gain).

WHY7. "Why does the per-class form

\sum_{i} p_{i} (1 - p_{i})

read as 'expected error per class', and why does it equal

1 - \sum p_{i}^{2}

Expand it:

\sum_{i} p_{i} (1 - p_{i}) = \sum_{i} p_{i} - \sum_{i} p_{i}^{2} = 1 - \sum_{i} p_{i}^{2}

(using

\sum_{i} p_{i} = 1

) — so the two forms are algebraically identical. Reading each term:

p_{i}

is the chance the item truly is class

i

;

(1 - p_{i})

is the chance a random guess is not class

i

; their product is class

i

's mistake probability, summed into total

G

Edge cases

EC1. "What is

G

for a node with a single class (

p = (1)

K = 1

)?"

G = 1 - 1^{2} = 0

. With only one possible label you can never guess wrong — perfectly pure, the degenerate lower bound.

EC2. "What happens to the maximum Gini

1 - \frac{1}{K}

as the number of classes

K \to \infty

It approaches 1 but never reaches it. Infinitely many equally-likely classes make a correct random guess vanishingly unlikely, so impurity nears its ceiling.

EC3. "A node has proportions

(0.99, 0.01)

— is it 'pure enough' to have

G \approx 0

Yes,

G = 1 - 0.9 9^{2} - 0.0 1^{2} \approx 0.0198

, very close to 0. Near-total dominance by one class makes misclassification rare, so Gini is nearly zero without being exactly zero.

EC4. "A split sends all samples into one child and zero into the other. What is

G_{split}

The empty child has weight

\frac{0}{N} = 0

and is ignored;

G_{split}

equals the full child's Gini = the parent's Gini. Gain is 0 — a useless split the tree won't take.

EC5. "Empty node (

N_{c} = 0

): does its

G_{c}

(which is

0/0

, undefined) break the split score?"

No. Its weight

\frac{N _{c}}{N} = 0

zeroes out the term before the undefined

G_{c}

matters. By convention an empty child contributes nothing to

G_{split}

EC6. "For a fixed 2-class node, where exactly does

G

peak as

p

moves from 0 to 1?"

p = 0.5

, giving

G = 0.5

. The curve

G = 2 p (1 - p)

(figure s01) is a downward parabola symmetric about

p = 0.5

, zero at both ends (pure) and maxed dead center.

Connections

Decision Trees — where these split scores are actually applied.
Entropy and Information Gain — the log-based sibling contrasted in WHY6, SE7.
Overfitting and Pruning — the trap behind WHY4.
Random Forests — why cheap Gini matters at scale (WHY6).
Classification vs Regression Trees — regression swaps Gini for variance/MSE.
Gini Coefficient (Economics) — the name-collision trap in SE5.