Loss functions - MSE, cross-entropy: Worked examples

This page is the exhaustive worked-examples companion to the parent note on MSE & cross-entropy. The parent told you what the formulas are and why they exist. Here we make sure you never meet a case you have not already solved by hand: every sign of error, the zero-error and worst-case extremes, both binary and multi-class, a real-world word problem, and an exam-style trap.

Before we compute anything, some reminders so no symbol is unearned:

Definition The three formulas we will hammer (all batch-averaged)

$\overset{y}{^}$ (read "y-hat") is the network's prediction; $y$ (plain "y") is the truth.
MSE (Mean Squared Error), used when the output is a real number: $MSE = \frac{1}{n} \sum_{i = 1}^{n} (y_{i} - \overset{y}{^}_{i})^{2}$ Here $n$ is the number of examples, $\sum$ ("sigma") means "add up over all $i$ ", and $()^{2}$ squares.
BCE (Binary Cross-Entropy) and CE (Categorical Cross-Entropy), used when the output is a probability. We write them per-example first, then average over the batch exactly like MSE — the same $\frac{1}{n} \sum$ on the front: $BCE = - \frac{1}{n} \sum_{i = 1}^{n} [y_{i} lo g \overset{y}{^}_{i} + (1 - y_{i}) lo g (1 - \overset{y}{^}_{i})]$ $CE = - \frac{1}{n} \sum_{i = 1}^{n} \sum_{k = 1}^{K} y_{i, k} lo g \overset{y}{^}_{i, k}$ $lo g$ here is the natural log ( $ln$ , base $e \approx 2.718$ ). $K$ is the number of classes. $y_{k}$ is one-hot: it is $1$ for the correct class and $0$ everywhere else. In every single-example computation below, $n = 1$ , so the $\frac{1}{n}$ is just $1$ and quietly disappears — but the same averaging is there whenever a batch appears (as in Cell D).

Common mistake Two consistency conventions to nail down NOW (so nothing is ambiguous later)

The $\frac{1}{2}$ on MSE. Some textbooks write $\frac{1}{2 n} \sum (y - \overset{y}{^})^{2}$ . That extra $\frac{1}{2}$ is a cosmetic choice: it cancels the $2$ that squaring produces when you differentiate, giving the tidy gradient $\overset{y}{^} - y$ . It does not change where the minimum is (a constant scale never moves an argmin). On this whole page we use NO $\frac{1}{2}$ — our MSE is exactly $\frac{1}{n} \sum (y - \overset{y}{^})^{2}$ , so its per-example gradient is $2 (\overset{y}{^} - y)$ , and we will carry that factor of $2$ honestly in Cell K.
The $lo g$ domain. $lo g \overset{y}{^}$ is only defined for $\overset{y}{^} > 0$ , and $lo g (1 - \overset{y}{^})$ only for $\overset{y}{^} < 1$ . So cross-entropy legally accepts only $\overset{y}{^} \in (0, 1)$ — never exactly $0$ or $1$ . That is the cliff you see in the figure: as $\overset{y}{^} \to 0$ the loss $\to + \infty$ . Real libraries clamp $\overset{y}{^}$ to $[ϵ, 1 - ϵ]$ (tiny $ϵ \approx 1 0^{- 7}$ ) so a confident-wrong output gives a big-but-finite number instead of NaN. See Softmax and Sigmoid for how we guarantee the output lands strictly inside $(0, 1)$ .

Intuition The one picture that unifies everything — how to read figure s01

The figure below has two panels, each a curve you will "read a value off" in every example.

Left panel (teal curve): the MSE bowl. Horizontal axis = the signed error $e = y - \overset{y}{^}$ ; vertical axis = the penalty $e^{2}$ . Notice it is a symmetric U — the two orange/plum dots mark the $0.5$ -miss and the $1.0$ -miss from Cell A/B/C, showing the $1.0$ miss sits four times higher.
Right panel (orange curve): the cross-entropy "surprise" cliff. Horizontal axis = the probability $\overset{y}{^}$ the model gave the true class; vertical axis = the loss $- lo g \overset{y}{^}$ . The three dots mark Cells E ( $0.9$ , near the floor), G ( $0.5$ , the dashed $ln 2$ line), and F ( $0.1$ , climbing the cliff). As $\overset{y}{^} \to 0$ the curve shoots to infinity — that is Cell F's " $\to \infty$ " and exactly the $lo g$ -domain cliff flagged above.

Keep those two shapes in your head — every example below is just reading a point off one of them.

Figure — Loss functions - MSE, cross-entropy

The scenario matrix

Every case this topic can throw at you falls into one of these cells. The examples that follow are labelled with the cell(s) they cover, and together they fill every row.

Cell	Loss	The scenario it stress-tests
A — negative error	MSE	prediction below truth ( $\overset{y}{^} < y$ )
B — positive error	MSE	prediction above truth ( $\overset{y}{^} > y$ )
C — zero error (degenerate)	MSE	perfect prediction, loss $= 0$
D — mixed batch	MSE	many points, signs mix, we average
E — confident & correct	BCE	$\overset{y}{^} \to 1$ when $y = 1$ : loss $\to 0$
F — confident & WRONG (limit)	BCE	$\overset{y}{^} \to 0$ when $y = 1$ : loss $\to \infty$
G — maximum uncertainty	BCE	$\overset{y}{^} = 0.5$ : the coin-flip baseline
H — the $y = 0$ branch	BCE	truth is the negative class
I — multi-class from logits	CE + softmax	raw scores $\to$ probs $\to$ loss
J — real-world word problem	MSE	units, scaling, interpretation
K — exam trap	any	why MSE-on-classification stalls

Cells A, B, C — one point, every sign of error (MSE)

Compute the raw errors $e = y - \overset{y}{^}$ .
- $A : 3 - 2.5 = + 0.5$ , $B : 3 - 4.0 = - 1.0$ , $C : 3 - 3 = 0$ . Why this step? The sign tells us the direction we under/over-shot. Notice A and B have opposite signs — squaring will erase that, which is exactly what we want for a "how wrong" number.
Square each error.
- $A : 0. 5^{2} = 0.25$ , $B : (- 1.0)^{2} = 1.0$ , $C : 0^{2} = 0$ . Why this step? Squaring makes every penalty positive and bends the penalty curve upward, so the $1.0$ miss costs $4 \times$ the $0.5$ miss — not $2 \times$ . That is the quadratic bowl in the figure above.

Verify: ratio of penalties $= 1.0/0.25 = 4$ , matching "twice as far ⇒ four times as bad". Cell C confirms the degenerate case: a perfect prediction has exactly zero loss, the bottom of the bowl. Units: (lakh)² — squared, which is why we usually quote MSE relative to a baseline, not as money.

Cell D — a mixed batch, averaged (MSE)

Square first, then average — never the other way. $MSE = \frac{1}{3} (0. 5^{2} + 0^{2} + (- 1.0)^{2}) = \frac{1}{3} (0.25 + 0 + 1.0) = \frac{1.25}{3}$ Why this step? If we averaged the raw errors we'd get $(0.5 + 0 - 1.0) /3 = - 0.167$ , which pretends the model is "almost right" — the $+ 0.5$ and $- 1.0$ mislead by cancelling. Squaring before summing kills all cancellation, so a batch of half-too-high and half-too-low still reports real error.
Divide by $n = 3$ . $MSE = 0.4166 \dots \approx 0.417$ Why this step? Dividing by $n$ makes the number comparable across batch sizes — a 3-house MSE and a 3000-house MSE live on the same scale. This is the same $\frac{1}{n}$ that sits on the front of BCE and CE too.

Verify: $1.25/3 = 0.41 \overline{6}$ . Sanity check: the mean loss must sit between the smallest per-point loss ( $0$ ) and the largest ( $1.0$ ) — and $0.417$ does. ✓

Cells E, F, G — binary cross-entropy across its whole range

Because $y = 1$ , the BCE formula collapses: $BCE = - [1 \cdot lo g \overset{y}{^} + 0 \cdot lo g (1 - \overset{y}{^})] = - lo g \overset{y}{^}$ . Only the first term survives.

Confident & correct (Cell E): $- lo g 0.9 = 0.1054$ . Why so small? The truth was highly probable under the model, so little surprise. On the surprise curve (right panel of the figure) $\overset{y}{^} = 0.9$ is far right, near the floor.
Confident & wrong (Cell F): $- lo g 0.1 = 2.3026$ . Why so large? The truth had only $10%$ probability — a nasty surprise. As $\overset{y}{^} \to 0$ , $- lo g \overset{y}{^} \to \infty$ : the loss is unbounded, punishing overconfidence savagely. (Recall $\overset{y}{^}$ is never exactly $0$ — libraries clamp it, so you get a big finite number, not NaN.)
Maximum uncertainty (Cell G): $- lo g 0.5 = 0.6931 = ln 2$ . Why this exact value? $\overset{y}{^} = 0.5$ means "I'm guessing." $ln 2 \approx 0.693$ is the famous coin-flip baseline — if your training BCE hovers here, the model has learned nothing.

Verify: ratio $2.3026/0.1054 \approx 21.8$ — over twenty times worse, far beyond most people's forecast. And $- lo g 0.5 = ln 2 = 0.6931$ . ✓ Limit check: try $\overset{y}{^} = 0.001$ , then $- lo g 0.001 = 6.9$ , still finite but growing without bound as we approach $0$ — confirming Cell F's " $\to \infty$ " claim.

Cell H — the OTHER branch: truth is the negative class

Substitute $y = 0$ into $BCE = - [y lo g \overset{y}{^} + (1 - y) lo g (1 - \overset{y}{^})]$ . $BCE = - [0 \cdot lo g 0.2 + 1 \cdot lo g (1 - 0.2)] = - lo g (0.8)$ Why this step? When $y = 0$ the first term is multiplied by zero and vanishes; the second term switches on. This is the mirror image of Cells E–G. Many people forget this branch exists — that is exactly why it is its own cell.
Evaluate. $- lo g (0.8) = 0.2231$ . Why this size? The model gave the true (negative) class $80%$ probability — fairly confident and correct, so a modest loss.

Verify: $- lo g 0.8 = 0.2231$ . Symmetry check: a $y = 0$ prediction of $\overset{y}{^} = 0.2$ must give the same loss as a $y = 1$ prediction of $\overset{y}{^} = 0.8$ , because both assign $0.8$ to the true class. Indeed $- lo g 0.8 = 0.2231$ either way. ✓

Cell I — multi-class, starting from raw logits (softmax + CE)

Exponentiate every logit. $e^{2} = 7.389, e^{1} = 2.718, e^{0.1} = 1.105$ . Why this step? $e^{z}$ is always positive and turns "bigger logit" into "much bigger weight" — this is the softmax numerator. We cannot take $lo g$ of a raw logit (it can be negative, and $lo g$ is undefined for negatives), so we must convert to probabilities in $(0, 1)$ first.
Sum and normalise. Sum $= 7.389 + 2.718 + 1.105 = 11.212$ . $\overset{y}{^} = [\frac{7.389}{11.212}, \frac{2.718}{11.212}, \frac{1.105}{11.212}] = [0.659, 0.242, 0.099]$ Why this step? Dividing by the sum forces the outputs to add to $1$ — now they are a valid probability distribution, the legal input to $lo g$ .
Pick the true-class term only. With one-hot $y = [1, 0, 0]$ , $CE = - lo g \overset{y}{^}_{0} = - lo g 0.659$ . Why only class 0? The one-hot zeros multiply away the other two terms — cross-entropy always reduces to $- lo g (prob of the correct class)$ .
Evaluate. $- lo g 0.659 = 0.4173$ .

Verify: probabilities sum to $0.659 + 0.242 + 0.099 = 1.000$ ✓, and $- lo g 0.659 = 0.4173$ . Contrast: if the model had instead given the true class $0.99$ , loss would be $- lo g 0.99 = 0.01$ — near zero, as a confident correct classifier should score. ✓

Cell J — real-world word problem (MSE with units)

Errors $y - \overset{y}{^} = [20 - 21, 22 - 19, 24 - 25, 28 - 30] = [- 1, + 3, - 1, - 2]$ . Why this step? Signs mix (over and under-forecast) — perfect illustration of why we must square before averaging.
Square: $[1, 9, 1, 4]$ . Sum $= 15$ . Why square? The $+ 3$ °C blunder now dominates ( $9$ of the $15$ ), which is the point: MSE cares most about the worst day.
Average: $MSE = 15/4 = 3.75$ (°C²). Why divide by 4? $n = 4$ days, keeps it per-day comparable.
RMSE: $3.75 = 1.9365$ °C. Why take the root? MSE lives in °C² — meaningless to a human. The square root drags it back into °C so we can say "typically off by about 1.94°C."

Verify: $15/4 = 3.75$ and $3.75 = 1.9365$ . Forecast check: RMSE ( $1.94$ °C) is below the worst single miss ( $3$ °C) but above the smallest ( $1$ °C) — RMSE always sits inside the range of absolute errors. ✓

Cell K — the exam trap: why MSE stalls on classification

First, one prerequisite we must not assume — the sigmoid:

We differentiate each loss with respect to the logit $z$ , using the chain rule $\frac{\partial L}{\partial z} = \frac{\partial L}{\partial y ^} \cdot σ^{'} (z)$ with $σ^{'} (z) = \overset{y}{^} (1 - \overset{y}{^})$ . The whole trick is what happens to that $\overset{y}{^} (1 - \overset{y}{^})$ factor in each case — so we show $\frac{\partial L}{\partial y ^}$ explicitly.

BCE branch — watch the factor cancel. With $y = 1$ , per-example $BCE = - lo g \overset{y}{^}$ , so $\frac{\partial BCE}{\partial y ^} = - \frac{1}{y ^} .$ Now multiply by $σ^{'} (z) = \overset{y}{^} (1 - \overset{y}{^})$ : $\frac{\partial BCE}{\partial z} = - \frac{1}{y ^} \cdot \overset{y}{^} (1 - \overset{y}{^}) = - (1 - \overset{y}{^}) = \overset{y}{^} - 1 = \overset{y}{^} - y .$ Why the cancellation happens: BCE's derivative carries a $\frac{1}{y ^}$ that exactly meets the $\overset{y}{^}$ inside $σ^{'}$ ; they annihilate, leaving the clean $\overset{y}{^} - y$ with no saturating factor left. (The general $y \in {0, 1}$ case gives the same $\overset{y}{^} - y$ — the $y = 0$ branch uses $\frac{\partial}{\partial y ^} [- lo g (1 - \overset{y}{^})] = \frac{1}{1 - y ^}$ , and the $(1 - \overset{y}{^})$ in $σ^{'}$ cancels it instead.)

MSE branch — the factor survives. With our no- $\frac{1}{2}$ MSE $= (\overset{y}{^} - y)^{2}$ for one point, $\frac{\partial MSE}{\partial y ^} = 2 (\overset{y}{^} - y) .$ Multiply by $σ^{'} (z) = \overset{y}{^} (1 - \overset{y}{^})$ : $\frac{\partial MSE}{\partial z} = 2 (\overset{y}{^} - y) \overset{y}{^} (1 - \overset{y}{^}) .$ Why nothing cancels: MSE's derivative has no $\frac{1}{y ^}$ to meet the $\overset{y}{^}$ in $σ^{'}$ — so the saturating factor $\overset{y}{^} (1 - \overset{y}{^})$ stays and throttles the gradient.

Now plug in $\overset{y}{^} = 0.02$ , $y = 1$ :

BCE gradient. $\overset{y}{^} - y = 0.02 - 1 = - 0.98$ . Why this size? Being badly wrong yields a large-magnitude push — exactly what learning needs.
MSE gradient. First the saturating factor: $\overset{y}{^} (1 - \overset{y}{^}) = 0.02 \times 0.98 = 0.0196$ . Then $2 (\overset{y}{^} - y) \overset{y}{^} (1 - \overset{y}{^}) = 2 (- 0.98) (0.0196) = - 0.038416.$ Why so tiny? The sigmoid is saturated near $0$ , so $\overset{y}{^} (1 - \overset{y}{^}) \approx 0.0196$ crushes the signal. Even the honest factor of $2$ can't rescue it — MSE tells a confidently-wrong model to barely change. Learning stalls.
Compare. $∣ - 0.98 ∣ = 0.98$ for BCE versus $∣ - 0.038416 ∣ = 0.038416$ for MSE. $\frac{∣ BCE grad ∣}{∣ MSE grad ∣} = \frac{0.98}{0.038416} \approx 25.51.$

Verify: ratio $0.98/0.038416 = 25.51$ — BCE sends a correction over twenty-five times stronger at this saturated point. That is precisely why we never use MSE for classification. ✓

Coverage check

Recall Did every cell get filled?

A,B,C ::: single-point MSE, both signs + zero (worked example A/B/C). D ::: averaged mixed batch (worked example D). E,F,G ::: BCE confident-correct, confident-wrong-limit, coin-flip (worked example E/F/G). H ::: BCE with $y = 0$ branch (worked example H). I ::: multi-class softmax from logits (worked example I). J ::: real-world temperature word problem with units/RMSE (worked example J). K ::: exam trap — MSE gradient saturation vs BCE, with full cancellation shown (worked example K).

See also: Maximum Likelihood Estimation (where both losses come from) and Regularization (what we add on top once the loss is chosen).