Cross-entropy concept: Exercises — AI-ML deep dive

This is a self-testing worksheet for the parent Cross-Entropy note. Every problem states its level (L1 Recognition → L5 Mastery). Try it first, then open the collapsible solution. Use natural log ( $ln$ , units = nats) throughout unless a problem says "bits" ( $lo g_{2}$ ).

Before we start, one reminder of every symbol we will use, in plain words:

Picture to hold in your head for the whole page — surprise as a curve:

Notice: as the predicted probability $q$ slides toward $0$ , the surprise $- lo g q$ shoots up to infinity. This single curve explains why confident-wrong predictions are punished so hard — that's the theme of the whole worksheet.

Level 1 — Recognition

Exercise 1.1

Which formula is the cross-entropy of $p$ relative to $q$ ? (a) $- \sum_{x} q (x) lo g p (x)$ (b) $- \sum_{x} p (x) lo g q (x)$ (c) $\sum_{x} p (x) lo g p (x)$ (d) $\sum_{x} (p (x) - q (x))^{2}$

Recall Solution

(b). Read it left to right: "weight by reality $p$ , take surprise of the guess $q$ ." Option (a) swaps the roles (that would be $H (q, p)$ , a different number). Option (c) is $- H (p)$ (wrong sign, and no $q$ ). Option (d) is mean squared error, not cross-entropy.

Exercise 1.2

True or false: cross-entropy can be negative for ordinary probability distributions (all $p (x), q (x) \in [0, 1]$ ).

Recall Solution

False. Each $q (x) \leq 1$ , so $lo g q (x) \leq 0$ , so $- lo g q (x) \geq 0$ . A sum of non-negative surprises weighted by non-negative $p (x)$ is $\geq 0$ . (For discrete distributions it can only reach $0$ when $q$ puts probability $1$ on the single outcome $p$ is certain of.)

Level 2 — Application

Exercise 2.1

A biased coin lands heads with true probability $p = 0.6$ . Your model guesses $q = 0.6$ (perfect). Compute $H (p, q)$ in nats and confirm it equals $H (p)$ .

Recall Solution

Two outcomes: heads $p = 0.6$ , tails $0.4$ . $H (p, q) = - [0.6 ln 0.6 + 0.4 ln 0.4] = - [0.6 (- 0.5108) + 0.4 (- 0.9163)]$ $= - [- 0.3065 - 0.3665] = 0.6730 nats.$ Because $q = p$ , the "inside the log" and "outside" numbers are identical, so this is $H (p)$ . The waste $D_{K L} = 0$ . This is the floor: no guess can beat a perfect one.

Exercise 2.2

Same true coin ( $p = 0.6$ heads), but now the model guesses $q = 0.9$ heads. Compute $H (p, q)$ and the wasted bits $D_{K L} (p ∥ q)$ .

Recall Solution

$H (p, q) = - [0.6 ln 0.9 + 0.4 ln 0.1] = - [0.6 (- 0.10536) + 0.4 (- 2.30259)]$ $= - [- 0.06322 - 0.92103] = 0.98425 nats.$ Waste: $D_{K L} = H (p, q) - H (p) = 0.98425 - 0.67301 = 0.31124$ nats. Reading it: the model is over-confident about heads. When tails shows up (40% of the time) it pays the huge $- ln 0.1 = 2.30$ surprise, dragging the average way above the floor.

Exercise 2.3

Three-class problem, one-hot truth $p = [0, 1, 0]$ (the answer is class 2). Model predicts $q = [0.2, 0.5, 0.3]$ . Compute the loss.

Recall Solution

One-hot truth zeroes every term except the true class: $H (p, q) = - [0 \cdot ln 0.2 + 1 \cdot ln 0.5 + 0 \cdot ln 0.3] = - ln 0.5 = 0.6931 nats.$ Only the probability the model gave to the correct class matters. See Categorical Cross-Entropy.

Level 3 — Analysis

Exercise 3.1

Soft (smoothed) labels. Truth is not one-hot: $p = [0.1, 0.8, 0.1]$ (a label-smoothed cat, see Label Smoothing). Model predicts $q = [0.1, 0.8, 0.1]$ (matches exactly). Compute $H (p, q)$ and $D_{K L} (p ∥ q)$ .

Recall Solution

$H (p, q) = - [0.1 ln 0.1 + 0.8 ln 0.8 + 0.1 ln 0.1]$ $= - [0.1 (- 2.30259) + 0.8 (- 0.22314) + 0.1 (- 2.30259)] = - [- 0.23026 - 0.17851 - 0.23026] = 0.63903 nats.$ Since $q = p$ , $D_{K L} = 0$ . Key lesson: unlike the one-hot case, the loss floor here is not $0$ — even a perfect model pays $H (p) = 0.639$ nats because the labels themselves are uncertain. This is exactly why label smoothing changes the minimum achievable loss.

Exercise 3.2

Show numerically that cross-entropy is not symmetric: with $p = [0.7, 0.3]$ and $q = [0.4, 0.6]$ , compute both $H (p, q)$ and $H (q, p)$ and confirm they differ.

Recall Solution

$H (p, q) = - [0.7 ln 0.4 + 0.3 ln 0.6] = - [0.7 (- 0.91629) + 0.3 (- 0.51083)] = 0.64140 + 0.15325 = 0.79465.$ Wait — sign care: $- 0.7 (- 0.91629) = + 0.64140$ , $- 0.3 (- 0.51083) = + 0.15325$ , sum $= 0.79465$ nats. $H (q, p) = - [0.4 ln 0.7 + 0.6 ln 0.3] = - [0.4 (- 0.35667) + 0.6 (- 1.20397)] = 0.14267 + 0.72238 = 0.86505 nats.$ $0.79465 \neq = 0.86505$ , so $H (p, q) \neq = H (q, p)$ . The two questions "surprised when reality is $p$ but I assumed $q$ " versus its swap are genuinely different.

Exercise 3.3

Using $H (p, q) = H (p) + D_{K L} (p ∥ q)$ , and the fact $D_{K L} \geq 0$ , prove that for a fixed truth $p$ the smallest possible cross-entropy is $H (p)$ , attained only when $q = p$ .

Recall Solution

$H (p)$ does not contain $q$ , so it is a constant as we vary the model. Then $H (p, q) = constant H (p) + \geq 0 D_{K L} (p ∥ q) .$ A constant plus a non-negative quantity is minimized by driving the non-negative part to its minimum, $0$ . And $D_{K L} (p ∥ q) = 0$ iff $q = p$ (Gibbs' inequality). Therefore $min_{q} H (p, q) = H (p)$ at $q = p$ . This is why minimizing cross-entropy pulls the model toward reality (equivalently, toward Maximum Likelihood Estimation).

Level 4 — Synthesis

Exercise 4.1

A logistic-regression model (see Logistic Regression) outputs $\overset{y}{^} = σ (z)$ for the positive class, and the true label is $y = 1$ . The per-sample loss is $L = - ln \overset{y}{^}$ . Derive $\frac{d L}{d z}$ and show it equals $\overset{y}{^} - y$ , where $σ (z) = \frac{1}{1 + e ^{- z}}$ and $σ^{'} (z) = σ (z) (1 - σ (z))$ .

Recall Solution

Chain rule, in three visible steps. What: $L = - ln \overset{y}{^}$ , so $\frac{d L}{d y ^} = - \frac{1}{y ^}$ . What: $\frac{d y ^}{d z} = σ^{'} (z) = \overset{y}{^} (1 - \overset{y}{^})$ . Combine: $\frac{d L}{d z} = - \frac{1}{y ^} \cdot \overset{y}{^} (1 - \overset{y}{^}) = - (1 - \overset{y}{^}) = \overset{y}{^} - 1 = \overset{y}{^} - y .$ Why it's beautiful: the messy $1/ \overset{y}{^}$ from the log exactly cancels the $\overset{y}{^}$ from the sigmoid derivative, leaving the clean prediction minus target. That cancellation is the real reason cross-entropy pairs so naturally with sigmoid/softmax — no vanishing gradient when the model is confidently wrong.

Exercise 4.2

Contrast the gradient signal. Suppose true $y = 1$ but $\overset{y}{^} = 0.01$ (confidently wrong). Compute the cross-entropy gradient magnitude $∣ \overset{y}{^} - y ∣$ and compare it to the MSE gradient magnitude for loss $L_{M S E} = (y - \overset{y}{^})^{2}$ with the same sigmoid, which is $∣2 (\overset{y}{^} - y) \overset{y}{^} (1 - \overset{y}{^}) ∣$ .

Recall Solution

Cross-entropy: $∣ \overset{y}{^} - y ∣ = ∣0.01 - 1∣ = 0.99$ . A strong push. MSE: $∣2 (0.01 - 1) (0.01) (0.99) ∣ = ∣2 (- 0.99) (0.0099) ∣ = 0.0196$ . The MSE gradient is roughly $0.02$ , about $50 \times$ weaker, because the $\overset{y}{^} (1 - \overset{y}{^})$ factor collapses toward $0$ exactly when the model saturates. Conclusion: MSE stalls on confident mistakes; cross-entropy keeps pushing. This is the quantitative version of Example 3 in the parent note.

Level 5 — Mastery

Exercise 5.1

Decompose a full loss. Truth $p = [0.2, 0.8]$ (soft label), model $q = [0.5, 0.5]$ (maximally unsure). Compute $H (p, q)$ , $H (p)$ , and $D_{K L} (p ∥ q)$ , and verify the identity $H (p, q) = H (p) + D_{K L}$ .

Recall Solution

$H (p, q) = - [0.2 ln 0.5 + 0.8 ln 0.5] = - ln 0.5 = 0.69315 nats.$ (Both classes get $q = 0.5$ , so the loss is just $- ln 0.5$ regardless of $p$ .) $H (p) = - [0.2 ln 0.2 + 0.8 ln 0.8] = - [0.2 (- 1.60944) + 0.8 (- 0.22314)] = 0.32189 + 0.17851 = 0.50040 nats.$ $D_{K L} = 0.69315 - 0.50040 = 0.19275 nats.$ Check the identity: $H (p) + D_{K L} = 0.50040 + 0.19275 = 0.69315 = H (p, q) .$ ✓ Interpretation: the model wastes $0.193$ nats by refusing to commit toward class 2, on top of the $0.500$ nats of irreducible label uncertainty.

Exercise 5.2

Optimal constant predictor. Over a large dataset the empirical class frequencies are $p = [0.25, 0.75]$ . If a model is forced to output the same $q = [q_{1}, 1 - q_{1}]$ for every sample, which $q_{1}$ minimizes the average cross-entropy, and what is the minimum value?

Recall Solution

By Exercise 3.3, average cross-entropy is minimized when $q = p$ . So $q_{1} = 0.25$ . Minimum loss $= H (p)$ : $H (p) = - [0.25 ln 0.25 + 0.75 ln 0.75] = - [0.25 (- 1.38629) + 0.75 (- 0.28768)]$ $= 0.34657 + 0.21576 = 0.56233 nats.$ Why this is the mastery insight: cross-entropy trained on data drives the model's output distribution toward the empirical class frequencies — that's the maximum-likelihood answer. Any $q_{1} \neq = 0.25$ adds a positive $D_{K L}$ .

Exercise 5.3

Bits vs nats. Convert the answer of Exercise 5.2 into bits.

Recall Solution

$H (p)_{bits} = \frac{0.56233}{l n 2} = \frac{0.56233}{0.69315} = 0.81128 bits.$ Same information content, different ruler. Nats use base $e$ ; bits use base $2$ ; divide nats by $ln 2$ to convert.

[!recall]- One-line self-check

Cross-entropy of a perfect model on soft labels equals

the Shannon entropy

H (p)

of the labels, not zero.

The gradient of

- ln \overset{y}{^}

through a sigmoid w.r.t. the logit is

\overset{y}{^} - y

To convert nats to bits you

divide by

ln 2 \approx 0.6931