Cross-entropy concept: Worked examples — AI-ML deep dive

This page is the "brute-force" companion to the parent note. We built the definition there. Here we hunt down every kind of input cross-entropy can face and work each one fully, so you never meet a case you have not seen.

Before anything, one reminder of the machine we are feeding numbers into, plus two supporting symbols we will lean on repeatedly.

Everything below is: pick $p$ , pick $q$ , weight surprise by reality, add up. The bar-chart figures show what $p$ (blue) and $q$ (orange) actually look like side by side.

The scenario matrix

Cross-entropy takes two probability vectors. The "scenarios" come from what those vectors can look like — hard vs soft truth, good vs bad guess, and the dangerous edges where a probability is $0$ or $1$ .

#	Cell (case class)	What makes it special	Example
A	Soft truth, soft guess, close	Both distributions spread out, model near-correct	Ex 1
B	Soft truth, soft guess, mismatched	Model confident in the wrong direction	Ex 2
C	Hard (one-hot) truth, decent guess	True vector like $[1, 0, 0]$ , model reasonable	Ex 3
D	Hard truth, confidently WRONG guess	The huge-penalty case	Ex 4
E	Degenerate: $q (x) = 0$ where $p (x) > 0$	$ln 0 = - \infty$ , the blow-up	Ex 5
F	Degenerate: $p (x) = 0$ term	$0 \cdot ln q$ — does it vanish?	Ex 5
G	Perfect match $q = p$ (lower bound)	Cross-entropy hits its floor $H (p)$	Ex 6
H	Real-world word problem	Translate a story into $p, q$	Ex 7
I	Exam twist: label smoothing	Truth is deliberately softened	Ex 8
J	Soft truth, HARD (one-hot) guess	Model commits to one class; finite unless it commits to a $p = 0$ class	Ex 8b
K	Limiting behaviour of a single term	How $- ln q$ grows as $q \to 0$ or $q \to 1$	Ex 9 (figure)

We now sweep every row.

Look at the bar chart: the blue truth bars and orange guess bars almost overlap — that visual near-coincidence is exactly why we expect the loss to sit barely above the floor.

Step 1 — Write out the surprises. $- ln (0.65) = 0.4308$ , $- ln (0.35) = 1.0498$ . Why this step? Surprise is per-outcome and uses the model's probability $q$ — that is the "code length" we committed to.

Step 2 — Weight by reality $p$ and sum. $H (p, q) = 0.7 (0.4308) + 0.3 (1.0498) = 0.3016 + 0.3149 = 0.6165 nats$ Why this step? Reality decides how often each surprise is actually incurred, so we weight by $p$ , not $q$ .

Verify. True entropy $H (p) = - [0.7 ln 0.7 + 0.3 ln 0.3] = 0.6109$ . Since $H (p, q) \geq H (p)$ always (the $D_{K L} \geq 0$ rule from the definition), and $0.6165 \geq 0.6109$ ✓. The gap $D_{K L} = 0.0056$ nats is tiny — matches "close guess". Units: nats ✓.

In the bar chart the orange guess is now flipped relative to blue truth — tall where truth is short. That crossing is the visual signature of a mismatched guess and warns us the loss will be large.

Step 1 — Surprises. $- ln (0.2) = 1.6094$ , $- ln (0.8) = 0.2231$ . Why this step? Same machine; only $q$ changed.

Step 2 — Weight by the true $p$ . $H (p, q) = 0.7 (1.6094) + 0.3 (0.2231) = 1.1266 + 0.0669 = 1.1935 nats$ Why this step? Rain happens 70% of the time, and every rainy day the model is very surprised — that heavy $0.7$ weight on the big surprise is what inflates the loss.

Verify. $H (p, q) = 1.19 ≫ 0.62$ from Ex 1 ✓. $D_{K L} = 1.1935 - 0.6109 = 0.5826$ nats — a large mismatch cost, as expected. Still $\geq H (p)$ ✓.

Notice in the chart: the blue truth is a single full bar on "cat" (that is what one-hot looks like). Every orange bar sitting over a zero blue bar will be multiplied away.

Step 1 — Kill the zero-weight terms. $H (p, q) = - [1 \cdot ln 0.7 + 0 \cdot ln 0.2 + 0 \cdot ln 0.1]$ Why this step? $p (dog) = p (bird) = 0$ , so those surprises never actually happen — their contribution is $0 \cdot (something finite) = 0$ .

Step 2 — Evaluate the one survivor. $H (p, q) = - ln 0.7 = 0.3567 nats$

Verify. For one-hot truth, $H (p) = 0$ (no uncertainty about a known cat), so $D_{K L} = H (p, q) = 0.3567$ — the whole loss is the mismatch. Sanity: a perfect $q (cat) = 1$ would give $- ln 1 = 0$ ✓. This is exactly the negative-log-likelihood shortcut and links to MLE.

Step 1 — Only the true class matters. $H (p, q) = - ln q (cat) = - ln 0.05$ Why this step? One-hot truth again zeroes the dog/bird terms — cross-entropy only reads the probability the model gave to what actually happened.

Step 2 — Evaluate. $H (p, q) = - ln 0.05 = 2.9957 nats$

Verify. Compare Ex 3 ( $0.357$ ) vs here ( $2.996$ ): dropping the true-class probability from $0.7$ to $0.05$ multiplied the loss ~8×. That steep penalty for confident wrongness is the whole reason we prefer cross-entropy over MSE. Note the model's confidence in "dog" ( $0.90$ ) never entered — cross-entropy doesn't reward being sure about the wrong thing ✓.

Sub-case E — $q$ zero where $p$ positive. Step 1. $H (p, q) = - ln q (cat) = - ln 0$ . Why this step? The true class occurs with probability $1$ , but the model swore it was impossible. Step 2. $- ln 0 = + \infty$ . Infinite loss. Why this matters: a model that rules out reality is infinitely surprised. This is why real code adds a tiny floor $ϵ$ (e.g. $q \to max (q, 1 0^{- 12})$ ) — or uses Label Smoothing so no probability is ever exactly $0$ .

Sub-case F — $p$ zero for a term. Here the dog term is $p (dog) ln q (dog) = 0 \cdot ln (0.2)$ . Since $ln (0.2)$ is a finite number ( $- 1.609$ ), the product is genuinely $0$ . Why this step? The danger $0 \cdot (- \infty)$ only appears when $q$ is also $0$ there. As long as $q (dog) > 0$ , the zero weight cleanly deletes the term. (By convention $0 ln 0 := 0$ , its limit.)

Verify. E: $lim_{q \to 0^{+}} (- ln q) = + \infty$ ✓. F: $0 \cdot ln 0.2 = 0$ exactly ✓. So the only dangerous zero is $q \to 0$ on the true class.

Step 1 — Plug in. $H (p, q) = - [0.6 ln 0.6 + 0.4 ln 0.4]$ Why this step? When $q = p$ the code lengths are already optimal for reality — zero wasted bits.

Step 2 — Evaluate. $- ln 0.6 = 0.5108$ , $- ln 0.4 = 0.9163$ . $H (p, q) = 0.6 (0.5108) + 0.4 (0.9163) = 0.3065 + 0.3665 = 0.6730 nats$

Verify. This equals Shannon Entropy $H (p)$ by construction, so $D_{K L} (p ∥ q) = 0$ ✓. And $0.673$ is a lower bound: no other $q$ can beat it (that is the $D_{K L} \geq 0$ rule again). This floor is why we can't drive classification loss to $0$ unless truth is one-hot ( $H (p) = 0$ ).

Step 1 — Translate the story to vectors. Truth $p = [0.3, 0.7]$ ; guess $q = [0.5, 0.5]$ . Here $N = 2$ (spam / ham). Why this step? "30% are truly spam" is the frequency — that's $p$ . "Model always says 50-50" is $q$ .

Step 2 — Surprises are equal. $- ln 0.5 = 0.6931$ for both classes. Why this step? A uniform guess assigns the same code length everywhere.

Step 3 — Weighted sum. $H (p, q) = 0.3 (0.6931) + 0.7 (0.6931) = 0.6931 nats$ Why this step? When $q$ is uniform over $N$ outcomes, every surprise is $- ln (1/ N) = ln N$ , and the $p$ -weights sum to $1$ , so $H (p, q) = ln N$ regardless of $p$ (here $ln 2$ ).

Verify. $ln 2 = 0.6931$ ✓. Compare $H (p) = - [0.3 ln 0.3 + 0.7 ln 0.7] = 0.6109$ . Wasted bits $D_{K L} = 0.0822$ nats — the price of refusing to commit. A smarter model outputting $q = p$ would save exactly that.

Step 1 — Note every term now counts. Why this step? This is exactly Pitfall 1 from the parent: soft $p$ means we can no longer drop terms. See Label Smoothing.

Step 2 — Surprises. $- ln 0.8 = 0.2231$ , $- ln 0.1 = 2.3026$ (twice). Why this step? Each surprise is the "how many nats does this outcome cost me" reading off the model's $q$ — and because $0.1$ is small, its surprise ( $2.30$ ) is large, ten-fold the surprise of the confident $0.8$ bar. The smaller the bar the taller the surprise; that inverse relationship is the whole geometry of $- ln$ .

Step 3 — Weighted sum. $H (p, q) = 0.9 (0.2231) + 0.05 (2.3026) + 0.05 (2.3026)$ $= 0.2008 + 0.1151 + 0.1151 = 0.4310 nats$ Why this step? The two small $0.05$ weights on the other classes now nudge the model to keep a little probability there — that is what label smoothing buys (calibration, less overconfidence).

Verify. One-hot loss on the same $q$ would be $- ln 0.8 = 0.2231$ . Smoothed loss $0.4310 > 0.2231$ ✓ — the extra $0.208$ comes from the two side terms. Floor check: $H (p)$ for $[0.9, 0.05, 0.05] = 0.3944$ , and $0.4310 \geq 0.3944$ ✓.

Step 1 — Near-one-hot guess, finite case. Surprises: $- ln 0.98 = 0.0202$ , $- ln 0.01 = 4.6052$ (twice). Why this step? A hard-ish guess makes the "wrong" surprises huge ( $4.6$ ), but the truth only weights them $0.05$ each — so they contribute but don't explode.

Step 2 — Weighted sum. $H (p, q) = 0.9 (0.0202) + 0.05 (4.6052) + 0.05 (4.6052) = 0.0182 + 0.2303 + 0.2303 = 0.4787 nats$ Why this step? Because $p$ still puts $0.05$ on each side class, over-confidence is penalised — a soft truth fights one-hot guesses. This is precisely why label smoothing discourages arrogant outputs.

Step 3 — The degenerate hard-wrong guess. If $q = [0, 1, 0]$ , the cat term is $p (cat) ln q (cat) = 0.9 \cdot ln 0 = - \infty$ , so $H (p, q) = + \infty$ . Why this step? Same rule as Cell E: a positive truth weight ( $0.9$ ) times $ln 0$ blows up. A one-hot guess is only safe when it lands exactly on classes the truth also cares about.

Verify. Finite case $0.4787 > 0.4310$ from Ex 8 (arrogance costs more than the mild guess) ✓, and $\geq H (p) = 0.3944$ ✓. Degenerate case: $lim_{q \to 0^{+}} 0.9 (- ln q) = + \infty$ ✓.

How to read this figure. The horizontal axis is $q$ , the probability your model assigns to the class that actually happened — it runs from $0$ (model swore it was impossible) to $1$ (model was fully certain and correct). The vertical axis is the resulting cross-entropy loss $L = - ln q$ in nats. Follow the blue curve left-to-right: it plunges from the top-left ( $+ \infty$ ) down to $0$ at the right edge. The three coloured dots mark the checkpoints below.

Step 1 — Read the endpoints. $q \to 1^{-}$ : $- ln q \to 0$ (perfectly confident and correct, no surprise). $q \to 0^{+}$ : $- ln q \to + \infty$ (confident and impossibly wrong — the Ex 5 blow-up). Why this step? These are the extreme corners every classifier lives between.

Step 2 — Read the middle checkpoints. $q = 0.5 \Rightarrow - ln 0.5 = 0.693$ ; $q = 0.1 \Rightarrow 2.303$ ; $q = 0.9 \Rightarrow 0.105$ . Why this step? Notice the curve is convex and steep on the left — the slope is $- 1/ q$ , so at $q = 0.1$ the gradient magnitude is $10$ , versus $1.11$ at $q = 0.9$ . That is the "strong gradient when confidently wrong" property from the parent's MSE comparison.

Verify. Slope $\frac{d}{d q} (- ln q) = - \frac{1}{q}$ : at $q = 0.1$ it is $- 10$ , at $q = 0.9$ it is $- 1.111$ ✓. Convexity ( $\frac{1}{q ^{2}} > 0$ ) guarantees a single minimum at $q = 1$ ✓. This is the geometric root of why cross-entropy trains fast — see Logistic Regression.

Recall Quick self-check

One-hot truth $[1, 0, 0]$ , model $[0.5, 0.3, 0.2]$ — what is the loss? ::: $- ln 0.5 = 0.6931$ nats (only the true-class term survives). Which single zero makes cross-entropy infinite? ::: $q = 0$ on a class where $p > 0$ (model rules out reality). If $q = p$ , cross-entropy equals what? ::: The Shannon entropy $H (p)$ — its minimum, with $D_{K L} = 0$ . Why does a uniform guess give $ln N$ regardless of $p$ ? ::: Every surprise equals $- ln (1/ N) = ln N$ , and the $p$ -weights sum to $1$ .

Related deep concepts: KL Divergence, Categorical Cross-Entropy, Mutual Information.