Entropy and information gain: Question bank — AI-ML deep dive

Every item below targets a specific misconception or boundary case. The four sections are True/False, Spot the error, Why questions, and Edge cases.

True or false — justify

Higher entropy always means a worse feature to split on.

False. Entropy of the parent is fixed; what matters is the weighted child entropy and the resulting information gain. A messy single child can still belong to a high-gain split.

If a child node has higher entropy than its parent, the split was invalid.

False. One child can exceed the parent, but the weighted average of all children never does — that is why gain is

\geq 0

by concavity.

Information gain can be negative for a badly chosen feature.

False. By concavity of

H

, the weighted child entropy never beats the parent, so the worst case is

I G = 0

, never negative.

Switching from

lo g_{2}

ln

changes which feature the tree picks.

False. Changing base multiplies every entropy by the same constant

1/ ln 2

, so it rescales all gains equally and the ranking of splits is unchanged. Only the numbers differ.

Entropy of

0

means the node has exactly one example.

False. It means the node is pure — every example shares one class. It could hold a thousand examples, all "Yes".

A node with 4 Yes and 4 No has the same entropy as one with 40 Yes and 40 No.

True. Entropy depends only on the proportions

p_{i}

, not the raw counts, so both are 50/50 giving

H = 1

bit.

Maximum binary entropy is always

1

regardless of log base.

False. It is

1

only with

lo g_{2}

. With

ln

the max is

ln 2 \approx 0.693

; the max is base-dependent even though the location

p = 0.5

is not.

Gini impurity and entropy always select the same split.

False. They usually agree because both are concave impurity measures, but on some distributions they disagree — they are different functions, only similarly shaped.

Information gain is unbiased across features with different numbers of values.

False. It is biased toward high-cardinality features, since splitting into many tiny pure groups inflates gain artificially.

Spot the error

H = - \sum p_{i} lo g_{2} p_{i}

, and since a pure node has

p = 1

for one class and

p = 0

for the rest, its entropy is undefined because

lo g_{2} 0

is undefined."

The

p = 0

terms use the convention

0 lo g_{2} 0 = 0

(its limit as

p \to 0

), so they vanish. The

p = 1

term gives

lo g_{2} 1 = 0

. Entropy is a clean

0

, not undefined.

"Gain = parent entropy − sum of child entropies."

Missing the weights. It must be parent entropy minus the weighted sum

\sum_{v} \frac{∣ S _{v} ∣}{∣ S ∣} H (S_{v})

; without weights you could get a negative or meaningless number.

"An ID column gives huge information gain, so it's the best feature to split on."

An ID splits into singletons, each perfectly pure, so gain is near-maximal but useless — it memorises rows and generalises to nothing. Use Gain Ratio to penalise this.

"Entropy uses a logarithm because logarithms happen to grow slowly."

The real reason is that

- lo g

is the only continuous function turning independent multiplication into addition:

I (pq) = I (p) + I (q)

. Slow growth is incidental, not the motivation.

"A split with the lowest weighted child entropy is not necessarily the highest-gain split."

This is actually correct — the parent entropy

H (S)

is the same constant for all candidate features, so lowest weighted child entropy is equivalent to highest gain. The claim as written is the error.

"Gini impurity ranges from 0 to 1, so it's on the same scale as binary entropy."

For two classes Gini maxes at

0.5

(at

p = 0.5

1 - 0.25 - 0.25

), not

1

. Only entropy hits

1

bit at 50/50; the scales differ.

"Since gain can be zero, a zero-gain split means the feature is random noise."

Not necessarily. Gain is

0

when every child has the same class proportions as the parent — the feature simply carries no information about the label here, which can be a property of this data, not proof of noise.

Why questions

Why do we weight child entropies by

∣ S_{v} ∣/∣ S ∣

instead of averaging them equally?

Because a random example lands in child

v

with probability

∣ S_{v} ∣/∣ S ∣

; the weighted sum is the expected leftover uncertainty. Big children influence the outcome more.

Why is information gain guaranteed to be non-negative?

Entropy is a concave function, so the weighted average of child entropies (evaluated at the child distributions) never exceeds the parent's entropy (evaluated at the mixed distribution) — Jensen's inequality forces

I G \geq 0

Why does Gini impurity exist if entropy already measures messiness?

Gini avoids the logarithm, making it cheaper to compute at scale. CART uses it; entropy and Gini usually agree on the split, so the speed is often free.

Why does the minus sign appear in the entropy formula?

Because probabilities satisfy

p \leq 1

, so

lo g_{2} p \leq 0

. The minus flips each term positive so entropy (a measure of positive uncertainty) comes out

\geq 0

Why does binary entropy peak exactly at

p = 0.5

At 50/50 you are maximally uncertain — every draw is a fair coin flip. Any imbalance makes one class predictable, lowering surprise, so entropy dips toward the pure ends.

Why is Gain Ratio preferred over raw gain in C4.5?

It divides gain by SplitInfo,

- \sum_{v} \frac{∣ S _{v} ∣}{∣ S ∣} lo g_{2} \frac{∣ S _{v} ∣}{∣ S ∣}

, which grows with the number of children, penalising high-cardinality features that inflate raw gain.

Why can cross-entropy loss in neural nets look like decision-tree entropy?

Both share the

- \sum p lo g q

shape — Cross-Entropy Loss measures surprise of predicted probabilities

q

under true labels

p

, the same "average surprise" idea entropy formalises. See KL Divergence for the general comparison.

Edge cases

A node has 100 examples, all "Yes". What is its entropy and what does a further split gain?

Entropy is

0

(pure). Any split leaves pure children, so gain is exactly

0

— there is no uncertainty left to remove.

A binary node is exactly 50/50. What is its entropy, and can any split reduce it?

Entropy is

1

bit (maximum). A split can reduce it — a perfect split into two pure children yields gain

1

; that is the dream case.

What happens to the term for a class with probability exactly

0

It contributes

0 lo g_{2} 0 = 0

by the limiting convention

lim_{p \to 0} p lo g_{2} p = 0

; absent classes never break or inflate the sum.

A feature has one value that every example shares (all in one child). What is its gain?

Gain is

0

. The single child is identical to the parent, so weighted child entropy equals parent entropy — no information is gained.

A three-class node is a perfect uniform

1/3, 1/3, 1/3

. Is its entropy

1

bit?

No. Maximum entropy for

c

classes is

lo g_{2} c

, so here it is

lo g_{2} 3 \approx 1.585

bits. The "1 bit max" rule is only for two classes.

An ID feature splits 50 examples into 50 singletons. What is its raw gain vs. its Gain Ratio?

Raw gain equals the full parent entropy (every child pure) — deceptively maximal. Gain Ratio is small because SplitInfo

= lo g_{2} 50

is large, correctly deflating the fake gain.

If two candidate features tie on information gain, what breaks the tie?

Gain alone cannot; algorithms fall back on Gain Ratio, Gini, or arbitrary/first-listed order. This is a genuine ambiguity, not a computed winner — see ID3 and C4.5 Algorithms.

Splitting always to pure leaves gives gain at every step — is that ideal?

No. Driving to pure leaves overfits, memorising noise. Pruning trades a little training purity for generalisation, per Overfitting and Pruning.

Connections

Decision Trees — the traps above all bite during node splitting.
Gini Impurity — the scale and agreement subtleties.
ID3 and C4.5 Algorithms — cardinality bias and Gain Ratio.
Cross-Entropy Loss — the shared $- \sum p lo g q$ shape.
KL Divergence — comparing distributions, entropy generalised.
Overfitting and Pruning — why pure leaves are not the goal.