Entropy and information gain: Worked examples

Everything below uses only these two formulas, restated so you never scroll up — and every symbol is defined in plain words first.

The scenario matrix

Every entropy/IG problem you will ever meet is one of these case classes. The grid below is a true matrix: rows = how pure the node is, columns = single node vs a split vs a trap. Every filled cell names the example that covers it.

purity ↓ \ situation →	Single node (just $H$ )	A split (compute $I G$ )	Special / edge case
Pure ( $p = 1$ )	Ex 1 (C1)	—	Ex 9 limit $p \to 0$ (C9)
Balanced ( $50/50$ )	Ex 2 (C2)	Ex 5 perfect split (C5)	—
Skewed binary	Ex 3 (C3)	Ex 8 word problem (C8)	—
Multi-class ( $c > 2$ )	Ex 4 (C4)	Ex 10 compare features (C10)	—
Any purity, bad split	—	Ex 6 useless split (C6)	Ex 7 ID trap + SplitInfo=0 (C7)

So the ten case classes map onto this grid: C1–C4 are single-node entropy of increasing complexity; C5–C6 are the two extremes of a split (all mess removed / none removed); C7 is the trap plus its degenerate denominator; C8 translates words; C9 probes the limit; C10 is the exam-style feature race. We now walk every cell.

Figure s01 — binary entropy curve. Alt text: a smooth lavender arch of

H (p)

against

p

; it touches zero at

p = 0

and

p = 1

(both marked "pure,

H = 0

" in mint) and peaks at a coral dot

p = 0.5, H = 1

bit. Its three landmarks: zero at both pure ends, a peak of 1 bit at 50/50, and a concave (upward-bulging) shape — that concavity is why gain can never be negative. Keep this picture in mind for every example below.

Ex 1 — Pure node (C1)

Ex 2 — Balanced binary, the maximum (C2)

Ex 3 — Skewed binary with ugly logs (C3)

Worked example The classic node: 9 "Yes", 5 "No" out of 14. Find

$H$ . Forecast: it leans Yes, so guess: closer to 1 or closer to 0.9?

Step 1 — probabilities. $p = 9/14 \approx 0.643$ , $q = 5/14 \approx 0.357$ . Why this step? $H$ needs the two class fractions; here the counts are $9$ and $5$ out of $14$ , so we divide to get $p$ and $q$ before touching any logarithm.

Step 2 — convert each $lo g_{2}$ . Use $lo g_{2} x = ln x / ln 2$ . $lo g_{2} (0.643) \approx - 0.637$ , $lo g_{2} (0.357) \approx - 1.486$ . Why this step? Calculators give $ln$ ; divide by $ln 2 \approx 0.693$ to get bits.

Step 3 — assemble. $H = - (0.643) (- 0.637) - (0.357) (- 1.486) \approx 0.410 + 0.531 = 0.940$ Why this step? Entropy is the sum of the per-class terms $- p_{i} lo g_{2} p_{i}$ ; we now add the two pieces we computed to get the total mess of the bag.

Verify: leans Yes but not purely → below the 1-bit ceiling, above 0. $0.940$ sits just left of the peak in Figure s01. ✅ (This is the parent note's baseline number.)

Ex 4 — Multi-class: the ceiling isn't 1 (C4)

Worked example A node has 3 classes, 2 examples each (2 Red, 2 Green, 2 Blue). Find

$H$ and its max. Forecast: more classes than binary — can $H$ exceed 1? What's the new ceiling?

Step 1 — probabilities. Each $p_{i} = 2/6 = 1/3$ . Why this step? Here $c = 3$ , so there are three fractions $p_{i}$ ; each class has $2$ of the $6$ rows, giving $p_{i} = 1/3$ . We need these before summing.

Step 2 — plug in. $H = - 3 \cdot (\frac{1}{3} lo g_{2} \frac{1}{3}) = - lo g_{2} \frac{1}{3} = lo g_{2} 3 \approx 1.585$ Why this step? All three per-class terms are identical, so the sum is just three copies of one term — that collapses neatly to $lo g_{2} 3$ .

Step 3 — state the rule. For $c$ equally-likely classes, $H_{m a x} = lo g_{2} c$ . Here $c = 3$ . Why this step? Generalising Step 2: whenever every class is equally likely, the identical terms always collapse to $lo g_{2} c$ , so this gives the entropy ceiling for any $c$ .

Verify: the Figure s01 curve was for $c = 2$ (max 1). With 3 classes the ceiling rises to $lo g_{2} 3 \approx 1.585 > 1$ . The famous "max entropy = 1 bit" is only the binary case. ✅

Figure s02 — the entropy ceiling grows as $lo g_{2} c$ . Alt text: butter-yellow bars, one per class-count

c

from 2 to 10, each labelled with its height

lo g_{2} c

; the bars climb from

1.00

c = 2

to about

3.32

c = 10

. The slow climb is why comparing raw entropies across different class-counts is unfair.

Ex 5 — Perfect split, gain equals parent (C5)

Worked example Parent: 3 Yes, 3 No. Split on attribute

$A$ with two values, giving children $S_{v_{1}} = {3 Yes}$ and $S_{v_{2}} = {3 No}$ . Find $I G$ . Forecast: the split perfectly separates classes — what's the biggest gain possible here?

Step 1 — parent entropy. The parent is 3:3 = 50/50, so $p = q = 0.5$ and $H (S) = 1$ . Why this step? $I G$ = parent mess $-$ leftover mess, so we always need the baseline $H (S)$ before the question is asked.

Step 2 — child entropies. Child $S_{v_{1}}$ is all Yes ( $p = 1$ ) and $S_{v_{2}}$ all No ( $p = 1$ ), so each has $H = 0$ . Why this step? The gain formula subtracts the entropy of each child $H (S_{v})$ , so we must measure how messy the two sub-bags are; a pure bag scores $0$ .

Step 3 — weighted child entropy. $\sum_{v} \frac{∣ S _{v} ∣}{∣ S ∣} H (S_{v}) = \frac{3}{6} (0) + \frac{3}{6} (0) = 0$ Why weight by $∣ S_{v} ∣/∣ S ∣$ ? A random example lands in child $v$ with probability $∣ S_{v} ∣/∣ S ∣$ , so the expected leftover mess is this weighted average — here both children are pure, so it is $0$ .

Step 4 — gain. $I G = 1 - 0 = 1$ bit. Why this step? Information gain is defined as baseline minus expected leftover; plugging in $H (S) = 1$ and weighted-child $= 0$ gives the reduction in uncertainty the split achieved.

Verify: when children are pure, gain equals the entire parent entropy — you removed all the mess. Maximum achievable gain for this node. ✅

Figure s03 — a perfect split empties the uncertainty bar. Alt text: a tall coral parent bar labelled "parent

H = 1

" splits (lavender arrow "split removes ALL mess") into two flat mint bars labelled "child

H = 0

"; the whole bar of uncertainty disappears.

Ex 6 — Useless split, gain exactly zero (C6)

Worked example Parent: 6 Yes, 6 No. Split on attribute

$B$ gives two children each with 3 Yes, 3 No. Find $I G$ . Forecast: each child looks exactly like the parent. Did we learn anything?

Step 1 — parent entropy. Parent is 6:6 = 50/50 → $H (S) = 1$ . Why this step? We need the baseline mess before the split to have something to subtract from.

Step 2 — child entropies. Each child is 3:3 = 50/50 → $H = 1$ each. Why this step? The gain formula uses each child's entropy $H (S_{v})$ ; both sub-bags are still perfectly balanced, so each is as messy as the parent.

Step 3 — weighted child entropy. $\frac{6}{12} (1) + \frac{6}{12} (1) = 1$ . Why weight by $∣ S_{v} ∣/∣ S ∣$ ? This is the expected leftover mess — the average child entropy weighted by how likely an example is to fall in each child; both are equal size and equally messy.

Step 4 — gain. $I G = 1 - 1 = 0$ bits. Why this step? Gain is baseline minus expected leftover; here they are identical, so the split removed nothing.

Verify: the split preserved the same mess in every child → zero information gained. This is the concavity floor: gain can equal 0 but never go negative. Steel-man from the parent note's mistake: never judge a split by a single child — judge by this final $I G$ . ✅

Ex 7 — High-cardinality trap + Gain Ratio (C7)

Worked example 4 examples: (Yes, No, Yes, No). Attribute

ID gives every row its own value $v \in {1, 2, 3, 4}$ , so 4 children $S_{v}$ of size 1. Compare raw $I G$ vs Gain Ratio. Forecast: each child is a single row → pure. Is this the "best" feature? Feels too good…

Step 1 — parent entropy. Parent is 2 Yes, 2 No → 50/50 → $H (S) = 1$ . Why this step? Again the baseline mess — we cannot compute gain without it.

Step 2 — child entropies. Each child is exactly 1 row, hence one class only, hence pure → $H = 0$ . Why this step? The gain formula subtracts each child's $H (S_{v})$ ; a one-row bag can never be mixed. $I G_{raw} = 1 - (4 \cdot \frac{1}{4} \cdot 0) = 1 bit (looks perfect!)$

Step 3 — compute SplitInfo (how much the attribute fragments the data): $SplitInfo (A) = - \sum_{v} \frac{∣ S _{v} ∣}{∣ S ∣} lo g_{2} \frac{∣ S _{v} ∣}{∣ S ∣} = - 4 \cdot \frac{1}{4} lo g_{2} \frac{1}{4} = lo g_{2} 4 = 2$ Why this tool? Raw IG rewards any attribute that shatters data into singletons. SplitInfo measures that shattering (it's just the entropy of the child-size distribution), so we can divide it out.

Step 4 — Gain Ratio. $GainRatio = \frac{I G}{SplitInfo} = \frac{1}{2} = 0.5$ Why this step? Dividing the raw gain by how badly the attribute fragmented the data penalises many-valued attributes, so a genuinely useful low-cardinality split can now out-score the ID trap.

Verify: raw IG screamed "1 bit, best ever!" but that split memorises rows and won't generalise (see Overfitting and Pruning). Gain Ratio halves it to $0.5$ , exposing the trap. This is why C4.5 uses Gain Ratio, not raw IG. ✅

Ex 8 — Word problem (C8)

Worked example "Out of 10 emails, 7 are spam and 3 are real. Filtering by the word

FREE splits them: 5 emails contain FREE (4 spam, 1 real); 5 do not (3 spam, 2 real). How many bits does the word FREE buy us?" Forecast: FREE leans spammy but isn't decisive — guess a smallish gain.

Step 1 — translate words to counts. Parent $S$ = 7 spam, 3 real. Attribute $A =$ "contains FREE" with values $v \in {yes, no}$ : child "FREE" $= 4 : 1$ , child "no FREE" $= 3 : 2$ . Why this step? Entropy and gain only eat counts/fractions, so we first turn the sentence into the parent bag and its two sub-bags $S_{v}$ .

Step 2 — parent entropy. $p = 0.7$ : $H (S) = - 0.7 lo g_{2} 0.7 - 0.3 lo g_{2} 0.3 \approx 0.360 + 0.521 = 0.881$ Why this step? This is the baseline mess before the FREE question — the thing $I G$ subtracts the leftover from.

Step 3 — child entropies. "FREE" (4:1, $p = 0.8$ ): $H = - 0.8 lo g_{2} 0.8 - 0.2 lo g_{2} 0.2 \approx 0.257 + 0.464 = 0.722$ . "no FREE" (3:2, $p = 0.6$ ): $H = - 0.6 lo g_{2} 0.6 - 0.4 lo g_{2} 0.4 \approx 0.442 + 0.529 = 0.971$ . Why this step? The gain formula subtracts each child's entropy $H (S_{v})$ , so we must measure how messy each sub-bag remains after the split.

Step 4 — weighted child + gain. $weighted = \frac{5}{10} (0.722) + \frac{5}{10} (0.971) = 0.361 + 0.485 = 0.847$ $I G = 0.881 - 0.847 \approx 0.034 bits$ Why this step? We weight each child by $∣ S_{v} ∣/∣ S ∣$ to get the expected leftover mess, then subtract it from the baseline — that difference is exactly the information the word FREE gives.

Verify: units are bits ✓; gain is positive but tiny — FREE alone is a weak spam signal. Sensible: plenty of real emails also say "free". ✅

Ex 9 — Limiting behaviour as $p \to 0$ (C9)

Worked example What happens to

$H (p) = - p lo g_{2} p - (1 - p) lo g_{2} (1 - p)$ as $p$ shrinks toward 0? Check $p = 0.1, 0.01, 0.001$ . Forecast: near-pure node → should mess vanish? Does it drop fast or crawl?

Step 1 — evaluate. $p = 0.1$ : $H \approx 0.469$ . $p = 0.01$ : $H \approx 0.0808$ . $p = 0.001$ : $H \approx 0.0114$ . Why these points? To watch the trend, not just the endpoint.

Step 2 — reason about the limit. As $p \to 0$ , the term $- p lo g_{2} p \to 0$ because $p$ (going to zero) beats $lo g_{2} p$ (going to $- \infty$ slowly). Why does the tool "limit" appear here? Because $lo g_{2} 0$ is undefined — we cannot just "plug in", we must ask what value $H$ approaches. The limit is $H \to 0$ .

Verify: this justifies the convention $0 lo g_{2} 0 = 0$ used everywhere on this page — it's not an arbitrary rule but the genuine limit. Numbers march $0.469 \to 0.081 \to 0.011 \to 0$ , hugging the left edge of the Figure s01 curve. ✅

Ex 10 — Exam twist: pick the winning feature (C10)

Worked example Parent: 14 examples, 9 Yes, 5 No (

$H = 0.940$ ). Two candidate attributes: Wind → Weak(6Y,2N) & Strong(3Y,3N). Outlook → Sunny(2Y,3N), Overcast(4Y,0N), Rain(3Y,2N). Which feature does a tree choose? Forecast: Outlook has a pure Overcast child — will that push it ahead of Wind?

Step 1 — Wind gain. Parent $H (S) = 0.940$ (Ex 3). Child entropies: Weak(6:2, $p = 0.75$ ) $H = 0.811$ ; Strong(3:3, 50/50) $H = 1.000$ . $weighted = \frac{8}{14} (0.811) + \frac{6}{14} (1.000) \approx 0.892, I G_{Wind} = 0.940 - 0.892 = 0.048$ Why this step? To choose a feature we need each candidate's gain; for Wind that means the baseline mess minus the expected leftover mess of its two children — so we compute both child entropies and weight them by child size.

Step 2 — Outlook child entropies. Sunny (2:3, $p = 0.4$ ): $H = - 0.4 lo g_{2} 0.4 - 0.6 lo g_{2} 0.6 \approx 0.971$ . Overcast (4:0): pure → $H = 0$ . Rain (3:2, $p = 0.6$ ): $H \approx 0.971$ . Why? Each of Outlook's three children needs its own mess measured before we can average them.

Step 3 — Outlook weighted child + gain. $weighted = \frac{5}{14} (0.971) + \frac{4}{14} (0) + \frac{5}{14} (0.971) \approx 0.347 + 0 + 0.347 = 0.694$ $I G_{Outlook} = 0.940 - 0.694 \approx 0.246 bits$ Why this step? Each child is weighted by $∣ S_{v} ∣/∣ S ∣$ to get the expected leftover mess; subtracting it from the baseline $0.940$ gives Outlook's gain — the pure Overcast child contributes $0$ mess.

Step 4 — compare. $0.246 > 0.048$ , so a tree splits on Outlook first. Why? The pure Overcast child (zero mess, real weight) does most of the cleanup.

Verify: both gains positive ✓ (concavity), and the feature with a pure child wins — exactly how Decision Trees grow their root. ✅

Figure s04 — Outlook vs Wind information gain. Alt text: two bars, a short lavender "Wind" bar labelled

0.048

bits beside a tall mint "Outlook" bar labelled

0.246

bits; Outlook towers over Wind, so the decision is visual once you plot it.

Recall

Recall Why can gain never be negative (seen across Ex 5–6)?

Entropy is concave, so the weighted average of child entropies never exceeds the parent's. Worst case the children mirror the parent (Ex 6) → $I G = 0$ .

Recall Ex 4 catch: is "max entropy = 1" always true?

No — that's binary only. For $c$ equal classes, $H_{m a x} = lo g_{2} c$ ( $= 1.585$ for 3 classes).

Recall Ex 7 catch: why did raw IG lie?

It rewards shattering data into singletons. Dividing by SplitInfo (Gain Ratio) cancels the fake reward, exposing the ID-column trap.

Recall What if SplitInfo = 0 in Gain Ratio?

Only single-valued attributes hit that; they split nothing, so both $I G$ and SplitInfo are $0$ and the attribute is simply excluded — the zero denominator never selects a split.

Connections

Parent: Entropy & Information Gain — formulas built here.
Decision Trees — Ex 10 is exactly root selection.
Gini Impurity — log-free cousin; rerun these examples with $1 - \sum p_{i}^{2}$ .
ID3 and C4.5 Algorithms — Gain Ratio (Ex 7) lives in C4.5.
Cross-Entropy Loss — same $- \sum p lo g q$ shape.
KL Divergence — compares two distributions, not one.
Overfitting and Pruning — the Ex 7 singleton split is textbook overfitting.