Entropy and KL divergence: Worked examples — AI-ML deep dive

This page is the workbook for the parent topic. We will not re-derive the formulas from scratch — instead we hunt down every kind of input these formulas can meet and work each one to the last decimal. Think of it as a checklist: after this page, no probability distribution should ever surprise you.

Before we start, let us re-anchor the two symbols we will use over and over, in plain words.

Everywhere below, $lo g$ without a base means $lo g_{2}$ (answers in bits) unless we say "nats", in which case it is the natural log $ln$ .

The scenario matrix

Here is the full landscape of cases a problem in this topic can hand you. Every worked example below is tagged with the cell(s) it covers, and together they fill the whole grid.

Cell	What makes it special	Covered by
A. Uniform	all outcomes equally likely → entropy hits its maximum $lo g_{2} n$	Ex 1
B. Skewed / biased	one outcome dominates → entropy shrinks	Ex 2
C. Degenerate (certain)	one probability $= 1$ , rest $= 0$ → entropy $= 0$ , and the $0 lo g 0$ trap	Ex 2, Ex 3
D. KL both directions	$D_{K L} (p ∥ q)$ vs $D_{K L} (q ∥ p)$ → asymmetry made concrete	Ex 4
E. KL degenerate	some $q (x) = 0$ where $p (x) > 0$ → divergence blows up to $\infty$	Ex 5
F. Identity	$p = q$ → KL $= 0$ (the sanity floor)	Ex 4 (check)
G. Cross-entropy = KL + H	one-hot label vs prediction → classification loss	Ex 6
H. Limiting behaviour	continuous knob $p \to 0$ or $p \to 1$ : where does entropy peak?	Ex 7
I. Word problem	real-world compression / surprise story	Ex 8
J. Exam twist	combine additivity + independence in one shot	Ex 9

The blue curve above (call it the binary entropy curve $H_{2} (p)$ ) is the map we will keep returning to. Read it left to right: at the edges $p = 0$ and $p = 1$ the curve touches the floor (certainty, cell C), and at the middle $p = 0.5$ it reaches the ceiling of $1$ bit (cell A). Cell B lives on the slopes; cell H is the whole shape of the curve.

Cell A — the uniform maximum

Worked example Example 1 — Fair four-sided die (Cell A)

A perfectly balanced tetrahedral die shows faces ${1, 2, 3, 4}$ , each with probability $\frac{1}{4}$ . Find $H (X)$ in bits.

Forecast: Guess before reading. Four equally likely outcomes — how many yes/no questions to find which face? Jot a number.

Write the sum: $H (X) = - \sum_{x = 1}^{4} \frac{1}{4} lo g_{2} \frac{1}{4} .$ Why this step? Entropy is just the average of the surprisals $- lo g_{2} p (x)$ . Here every term is identical, so the average equals one term.
Each surprisal: $- lo g_{2} \frac{1}{4} = - (- 2) = 2$ bits. Why this step? $\frac{1}{4} = 2^{- 2}$ , and $lo g_{2} 2^{- 2} = - 2$ . Two bits = two yes/no questions ("is it in ${1, 2}$ ?" then "which of the two?").
Average: $H (X) = 4 \times \frac{1}{4} \times 2 = 2$ bits. Why this step? All four weighted terms are $\frac{1}{4} \cdot 2$ ; they sum to $2$ .

Verify: The maximum-entropy rule says $H \leq lo g_{2} n = lo g_{2} 4 = 2$ , hit with equality because the die is uniform. On the curve figure, four outcomes generalise the peak: uniform always sits at the ceiling. ✓

Cells B & C — skew and the zero-probability trap

Worked example Example 2 — Loaded coin, then a stuck coin (Cells B, C)

(B) A coin with $p (H) = 0.9, p (T) = 0.1$ . (C) A trick coin that always lands heads: $p (H) = 1, p (T) = 0$ . Find both entropies in bits.

Forecast: Will the loaded coin's entropy be closer to $1$ bit or to $0$ ? And what does the stuck coin do to the term $0 lo g_{2} 0$ ?

Loaded coin surprisals: $- lo g_{2} 0.9 = 0.152$ , $- lo g_{2} 0.1 = 3.322$ bits. Why this step? Rare events (the tail) carry more surprise — that is the whole point of the $- lo g$ shape.
Weighted average: $H = 0.9 (0.152) + 0.1 (3.322) = 0.137 + 0.332 = 0.469$ bits. Why this step? We weight each surprise by how often it actually happens; the rare-but-shocking tail is discounted by its rarity.
Stuck coin (C): $H = - [1 \cdot lo g_{2} 1 + 0 \cdot lo g_{2} 0]$ . First term $= 0$ since $lo g_{2} 1 = 0$ . Second term uses the convention $0 lo g_{2} 0 := 0$ . Why this step? $lo g_{2} 0$ is $- \infty$ , but it is multiplied by a probability of $0$ . The limit $lim_{p \to 0^{+}} p lo g_{2} p = 0$ (the outcome never happens, so it can't add surprise). So $H = 0$ .

Verify: $0.469$ sits on the slope of the blue curve between the $p = 0.9$ tick and the ceiling — below $1$ , above $0$ , exactly as skew predicts. The stuck coin lands on the floor at $p = 1$ . ✓

Cell C again — the true floor

Cells D & F — KL both directions and the identity floor

Worked example Example 4 — Truth vs model, and back again (Cells D, F)

True coin $p = (0.5, 0.5)$ . Model coin $q = (0.7, 0.3)$ . Compute $D_{K L} (p ∥ q)$ , then $D_{K L} (q ∥ p)$ , in bits. Also confirm $D_{K L} (p ∥ p) = 0$ .

Forecast: Will the two directions give the same number? Predict yes or no before computing.

$D_{K L} (p ∥ q) = 0.5 lo g_{2} \frac{0.5}{0.7} + 0.5 lo g_{2} \frac{0.5}{0.3}$ . Why this step? KL weights the log-ratio $lo g_{2} \frac{p}{q}$ by the true frequency $p (x)$ — we average the surprise of our wrongness under reality.
Compute ratios: $lo g_{2} \frac{0.5}{0.7} = lo g_{2} 0.714 = - 0.485$ ; $lo g_{2} \frac{0.5}{0.3} = lo g_{2} 1.667 = 0.737$ .
$D_{K L} (p ∥ q) = 0.5 (- 0.485) + 0.5 (0.737) = - 0.243 + 0.369 = 0.126$ bits. Why this step? Individual terms can be negative, but the total KL can never be — Gibbs' inequality guarantees $\geq 0$ .
Now flip. $D_{K L} (q ∥ p) = 0.7 lo g_{2} \frac{0.7}{0.5} + 0.3 lo g_{2} \frac{0.3}{0.5} = 0.7 (0.485) + 0.3 (- 0.737) = 0.340 - 0.221 = 0.119$ bits. Why this step? Now the weights are $q (x)$ , not $p (x)$ — a different average, hence a different number.
Identity check: $D_{K L} (p ∥ p) = \sum p lo g_{2} \frac{p}{p} = \sum p lo g_{2} 1 = 0$ .

Verify: $0.126 \neq = 0.119$ , so KL is asymmetric — never treat it as a distance. Both are strictly positive (models differ), and $p ∥ p$ gives exactly the floor $0$ . ✓

Cell E — the divergence explodes

Worked example Example 5 — When the model rules something out (Cell E)

True $p = (0.5, 0.5)$ over ${H, T}$ , but an overconfident model says $q = (1, 0)$ — "heads is impossible to lose". Compute $D_{K L} (p ∥ q)$ .

Forecast: The model assigns probability $0$ to something that happens half the time. Finite waste, or catastrophe?

Term for $T$ : $0.5 lo g_{2} \frac{0.5}{0} = 0.5 lo g_{2} (\infty) = + \infty$ . Why this step? Here $p (T) = 0.5 > 0$ but $q (T) = 0$ . The convention $D_{K L} = \infty$ when $p > 0, q = 0$ fires. Intuitively: to encode an event your code deemed impossible costs infinitely many bits.
So $D_{K L} (p ∥ q) = \infty$ , regardless of the heads term. Why this step? One infinite term dominates the whole sum.

Verify: Contrast with cell D where every $q (x) > 0$ gave a finite answer. The lesson for ML: never let a model output a hard $0$ probability (hence label smoothing and softmax, which keep all $q (x) > 0$ ). ✓

Cell G — cross-entropy, KL, and classification loss

Worked example Example 6 — One-hot label vs prediction (Cell G)

True label (class 2 of 3): $p = [0, 1, 0]$ . Model prediction $q = [0.1, 0.7, 0.2]$ . Find the cross-entropy loss $H (p, q)$ in nats, and show it equals $D_{K L} (p ∥ q)$ here.

Forecast: Because $p$ is one-hot, how many terms of the sum actually survive?

$H (p, q) = - \sum_{x} p (x) ln q (x) = - [0 \cdot ln 0.1 + 1 \cdot ln 0.7 + 0 \cdot ln 0.2]$ . Why this step? Cross-entropy scores the model by the surprise it assigns to the true outcome, weighted by the true distribution. A one-hot $p$ zeroes out every wrong-class term.
Survivor: $H (p, q) = - ln 0.7 = 0.357$ nats. Why this step? Only class 2 has $p > 0$ , so only $- ln q (2)$ remains — the negative log-likelihood of the correct class.
Link to KL: $D_{K L} (p ∥ q) = H (p, q) - H (p)$ . Here $H (p) = - 1 \cdot ln 1 = 0$ (one-hot is certain). Why this step? A one-hot label has zero entropy, so minimizing cross-entropy is minimizing KL — the constant $H (p)$ drops out during training.
Therefore $D_{K L} (p ∥ q) = 0.357 - 0 = 0.357$ nats.

Verify: Cross-entropy $\geq H (p) = 0$ always, and equals $0$ only for a perfect $q (2) = 1$ . Our $0.357$ nats is the penalty for the model spreading $30%$ mass onto the wrong classes. This is exactly the Cross-Entropy Loss used to train classifiers. ✓

Cell H — limiting behaviour, where the peak lives

Worked example Example 7 — Turning the bias knob (Cell H)

For a coin with $p (H) = p$ , the binary entropy is $H_{2} (p) = - p lo g_{2} p - (1 - p) lo g_{2} (1 - p)$ . Where is it maximal, and what are the limits as $p \to 0$ and $p \to 1$ ?

Forecast: Look at the blue curve in figure s01. Point to its highest spot and its two endpoints before computing.

Endpoints: as $p \to 0^{+}$ , $- p lo g_{2} p \to 0$ and $- (1 - p) lo g_{2} (1 - p) \to 0$ , so $H_{2} \to 0$ . Same at $p \to 1^{-}$ . Why this step? Both edges are near-certainty (almost always tails, or almost always heads) — no surprise, the curve kisses the floor.
Maximum: by symmetry and the concavity from the parent note, the peak is at $p = 0.5$ , giving $H_{2} (0.5) = - 0.5 lo g_{2} 0.5 - 0.5 lo g_{2} 0.5 = 1$ bit. Why this step? Maximum uncertainty = maximally spread distribution = the uniform case (cell A) for $n = 2$ .

Verify: $H_{2} (0.5) = 1$ matches the $lo g_{2} n = lo g_{2} 2 = 1$ ceiling. The curve is symmetric about $0.5$ because relabeling heads↔tails cannot change how uncertain you are. ✓

Cell I — a word problem

Worked example Example 8 — Compressing weather forecasts (Cell I)

A town's daily weather is $p = (sun 0.7, rain 0.2, snow 0.1)$ . A lazy engineer compresses using a uniform codebook $q = (\frac{1}{3}, \frac{1}{3}, \frac{1}{3})$ . (a) What is the true entropy $H (p)$ ? (b) How many extra bits per day does the uniform code waste, i.e. $D_{K L} (p ∥ q)$ ?

Forecast: The uniform code treats snow as often as sun — will it waste a lot or a little?

Entropy: $H (p) = - [0.7 lo g_{2} 0.7 + 0.2 lo g_{2} 0.2 + 0.1 lo g_{2} 0.1]$ .
- $0.7 lo g_{2} 0.7 = 0.7 (- 0.5146) = - 0.3602$
- $0.2 lo g_{2} 0.2 = 0.2 (- 2.3219) = - 0.4644$
- $0.1 lo g_{2} 0.1 = 0.1 (- 3.3219) = - 0.3322$ So $H (p) = 0.3602 + 0.4644 + 0.3322 = 1.157$ bits. Why this step? This is the best possible average code length for this weather.
KL to uniform: $D_{K L} (p ∥ q) = \sum_{x} p (x) lo g_{2} \frac{p ( x )}{1/3} = \sum_{x} p (x) (lo g_{2} p (x) + lo g_{2} 3)$ . Why this step? $lo g_{2} \frac{p}{1/3} = lo g_{2} p + lo g_{2} 3$ ; the constant $lo g_{2} 3$ pulls out of the sum.
Since $\sum_{x} p (x) = 1$ : $D_{K L} = lo g_{2} 3 - H (p) = 1.585 - 1.157 = 0.428$ bits. Why this step? $\sum_{x} p (x) lo g_{2} p (x) = - H (p)$ , so the whole thing collapses neatly. The KL to the uniform code always equals $lo g_{2} n - H (p)$ .

Verify: The lazy code costs $1.157 + 0.428 = 1.585 = lo g_{2} 3$ bits/day — exactly the cost of encoding 3 equally-likely symbols, which is what the uniform code pretends the weather is. The $0.428$ -bit waste is positive, as every KL must be. ✓

Cell J — the exam twist

Worked example Example 9 — Two independent dice at once (Cell J, additivity)

You roll the fair four-sided die from Ex 1 and flip the fair coin from the parent note, independently. What is the joint entropy $H (X, Y)$ ?

Forecast: Independent sources of surprise — do their entropies add, multiply, or something messier?

Independence means $p (x, y) = p (x) p (y)$ , so $lo g_{2} p (x, y) = lo g_{2} p (x) + lo g_{2} p (y)$ . Why this step? The log turns the product of independent probabilities into a sum of surprisals — the additivity property that made $lo g$ the right tool in the first place.
Therefore $H (X, Y) = H (X) + H (Y) = 2 + 1 = 3$ bits. Why this step? Expectation is linear, so the average of a sum of surprisals is the sum of averages. The die contributes $2$ bits (Ex 1), the coin $1$ bit.
Sanity via the joint uniform: the pair has $4 \times 2 = 8$ equally likely outcomes, and $lo g_{2} 8 = 3$ . Why this step? Two independent uniforms make one big uniform, whose entropy is $lo g_{2} (total outcomes)$ .

Verify: $H (X) + H (Y) = 3 = lo g_{2} 8$ . Additivity holds only because they are independent; if the coin depended on the die, $H (X, Y) < H (X) + H (Y)$ — the gap is the Mutual Information. ✓

Where this leads

The waste-bits idea generalises: KL is one member of the F-divergences family, and its symmetrized cousin is Jensen-Shannon Divergence.
Choosing the distribution of maximum entropy subject to constraints is the Maximum Entropy Principle.
In deep generative models, minimizing a KL term against a prior is the heart of Variational Autoencoders and the Evidence Lower Bound (ELBO).
Decision trees split on the entropy drop called Information Gain.

Recall Self-test (reveal after answering)

Fair 4-sided die entropy ::: $2$ bits ( $= lo g_{2} 4$ ) Loaded coin $p (H) = 0.9$ entropy ::: $0.469$ bits $D_{K L} (p ∥ q)$ for $p = (.5, .5), q = (.7, .3)$ ::: $0.126$ bits Same but reversed $D_{K L} (q ∥ p)$ ::: $0.119$ bits (asymmetric!) KL when $q$ assigns $0$ to an event with $p > 0$ ::: $+ \infty$ Cross-entropy of $p = [0, 1, 0]$ , $q = [0.1, 0.7, 0.2]$ ::: $- ln 0.7 = 0.357$ nats Weather $H (p)$ for $(0.7, 0.2, 0.1)$ ::: $1.157$ bits KL of that weather to uniform ::: $0.428$ bits Joint entropy of fair 4-die + fair coin ::: $3$ bits