Foundations — Gini impurity

This page assumes you know nothing. Before you can read the parent note's formula $G=1-{\sum }_{i=1}^K{p}_i^2$, you must first know what a proportion is, what that stretched-out S ($\sum$) means, why we square, and what a "node" even looks like. We build every one of those, in order, with pictures.

0. The picture the whole topic lives in

Everything in this topic is about sorting things into boxes. Look at the figure: a big box of mixed circles gets split into two smaller boxes. Each box is a node.

We need this word because Gini impurity is always computed for one node at a time. There is no "Gini of the whole tree" — only the Gini of each box.

1. Counting: $N$ and $N_c$

Before proportions, we must count.

Why the topic needs it: when we compare splits, a big child and a tiny child should not count equally. Counting lets us weight them (Section 6).

2. Class and proportion: $i$, $K$, and $p_i$

The circles come in colors. In machine learning a "color" is called a class — the label we're trying to predict (apple/orange, spam/not-spam, +/−).

Now the star symbol of the whole topic:

Two facts about proportions that we'll lean on:

• Every $p_i$ is between $0$ and $1$ (you can't have a negative share or more than 100%).
• They all add up to 1, because every item belongs to exactly one class. This is written ${\sum }_i{p}_i=1$ — which needs the next symbol.

3. The summation sign $\sum$

This stretched letter scares beginners. It is just "add up a list."

Why the topic needs it: the Gini formula must combine all the classes into one number. $\sum$ is the tool that says "loop over every class and total it up."

4. Squaring: $p_i^2$ and why we square

Look at the curve in the figure: plotting $p^2$ against $p$ bends downward. A big share (say $0.9$) barely shrinks ($0.81$); a small share ($0.1$) collapses ($0.01$).

5. Putting it together: reading $G=1-\sum p_i^2$

You now own every symbol. Read the parent's formula left to right:

Quick sanity checks (all cases):

• All one class: $p=(1,0)$, $\sum p_i^2=1$, so $G=1-1=0$. Pure box → never wrong. ✔
• Fifty-fifty (2 classes): $p=(0.5,0.5)$, $\sum =0.25+0.25=0.5$, $G=0.5$. Worst 2-class mix. ✔
• Equal thirds (3 classes): $\sum =3\times (1/3{)}^2=1/3$, $G=2/3$. Worst 3-class mix. ✔

6. Weighting a split: $N_c/N$

A split makes several children. To score it we need one number, so we average the children's Ginis — but weighted by how many items each holds.

The tree then picks the split with the smallest $G_{\text{split}}$, i.e. the biggest Gini gain $=G_{\text{parent}}-G_{\text{split}}$.

Prerequisite map

Equipment checklist

Test yourself — reveal only after you've answered aloud.

Connections

• Gini impurity — the parent topic this page prepares you for.
• Decision Trees — where nodes, splits and leaves come from.
• Entropy and Information Gain — same $p_i$ and $\sum$, different combining rule.
• Classification vs Regression Trees — classes vs continuous targets.
• Random Forests — many Gini-grown trees averaged.
• Gini impurity (Hinglish) — the same idea in Hinglish.