Visual walkthrough — Loss functions - MSE, cross-entropy

---

Step 1 — What is a prediction, really?

WHAT. A classifier does not shout "cat!". It hands you a list of confidences — one number per possible answer. If the choices are {cat, dog, fox}, it might say $[0.7,0.2,0.1]$.

WHY these three numbers and not one. A single "cat-ness" number can't say how the remaining belief is split between dog and fox. We need a full probability distribution: a set of numbers that (a) are each between $0$ and $1$, and (b) sum to exactly $1$ (all your belief has to go somewhere).

PICTURE. Think of a jar holding exactly one litre of water. You pour it into three cups labelled cat / dog / fox. The height in each cup is that class's probability. No water is created or destroyed — that's the "sums to 1" rule.

---

Step 2 — What is the truth we compare against?

WHAT. The true label is also written as a list, but a brutally simple one: a $1$ in the correct cup, $0$ everywhere else. If the real animal is a cat, the truth is $[1,0,0]$.

WHY write truth as a list too. So truth and prediction live in the same shape — three numbers vs three numbers — and we can compare them term by term. A list that is all $0$ except a single $1$ is called one-hot.

PICTURE. The truth is a jar poured entirely into one cup: full cat cup, empty dog and fox cups.

---

Step 3 — Turning "how good?" into "how surprised?"

WHAT. We need one number saying how wrong the prediction was. The trick: measure surprise. If the true answer was cat and you gave cat a probability near $1$, you're barely surprised. If you gave cat a probability near $0$, you're stunned.

WHY the function $-\log$ and not, say, $1-\hat{y}$. We want surprise to have three properties:

1. Certain truth (${\hat{y}}_t=1$) → zero surprise.
2. Impossible truth (${\hat{y}}_t\to 0$) → surprise blows up to $+\infty$ (a confident-and-wrong model must be punished savagely).
3. Independent evidence should add: seeing two unrelated correct facts should sum their surprises, not multiply. The only function turning the multiplication of probabilities into an addition of scores is the logarithm.

The plain function $1-\hat{y}$ fails property 2 — it caps out at $1$, so being catastrophically wrong costs the same as being mildly wrong. Only $-\log$ punishes near-zero probability without mercy.

PICTURE. Plot $-\log ({\hat{y}}_t)$ as the true-class probability slides from $0$ to $1$. Watch it start at $0$ on the right and rocket to the sky on the left.

---

Step 4 — Why the sum collapses to a single term

WHAT. Cross-entropy for one example is $\text{CE}=-{\sum }_{k=1}^K\underset{\begin{array}{c}\text{truth for}\\ \text{class }k\end{array}}{\underbrace{y_k}}\underset{\begin{array}{c}\text{log of predicted}\\ \text{prob for class }k\end{array}}{\underbrace{\log {\hat{y}}_k}}$ It looks like all $K$ classes contribute. They don't.

WHY it collapses. Every $y_k$ is either $0$ or $1$. A term with $y_k=0$ is $0\times (\text{anything})=0$ — it vanishes. Only the true class, where $y_t=1$, survives: $\text{CE}=-\log {\hat{y}}_t.$ So the general formula and the "just the surprise of the true class" idea from Step 3 are the exact same thing. The sum is only there to write it in one line for any label.

PICTURE. Line up the two lists. Multiply cup by cup. Two products are zeroed out; one survives — the true-class surprise from Step 3.

---

Step 5 — Where do the ${\hat{y}}_k$ come from? Softmax

WHAT. The network's last layer spits out raw scores called logits, written $z_k$. They can be any real number — negative, huge, whatever. They are not probabilities (they don't sum to 1, can be negative). Softmax converts them: ${\hat{y}}_k=\frac{e^{z_k}}{{\sum }_{j=1}^K{e}^{z_j}}.$

WHY exponentiate then divide. Two jobs, one formula:

• $e^{z_k}$ is always positive, no matter how negative $z_k$ is — that fixes rule (a) from Step 1.
• Dividing by the total ${\sum }_j{e}^{z_j}$ forces the outputs to sum to 1 — that fixes rule (b).

Exponentiation also makes the biggest logit dominate smoothly, which is exactly the "soft" version of picking the max.

PICTURE. Three logit bars go in; $e^{(\cdot )}$ stretches them (big ones stretch far more); dividing by the total rescales them into three probability bars that fill one unit of height.

See Softmax and Sigmoid for the full treatment of this squashing step.

---

Step 6 — The payoff: the gradient becomes ${\hat{y}}_k-y_k$

WHAT. Gradient Descent needs the slope of CE with respect to each logit $z_k$ — that slope tells Backpropagation which way to nudge the score. The remarkable result is: $\frac{\partial \text{CE}}{\partial z_k}={\hat{y}}_k-y_k.$ Prediction minus truth. Nothing else.

WHY it's remarkable and why it happens. The $-\log$ in CE has a derivative $-\frac{1}{\hat{y}}$ — it explodes near $0$. The softmax has a derivative carrying a $\hat{y}(1-\hat{y})$-type factor that shrinks near $0$. When you chain-rule them together, the exploding piece and the shrinking piece cancel exactly, leaving the tame, honest error ${\hat{y}}_k-y_k$.

Contrast this with sigmoid-plus-MSE (see the parent's mistake box): there is no $-\log$ to cancel the shrinking factor, so the gradient dies when the model is confidently wrong — learning stalls.

PICTURE. Two curves over the true-class probability: MSE's gradient magnitude (flattens to $0$ at both ends — the danger zone on the left) versus CE's gradient magnitude (stays strong and large exactly where the model is confidently wrong).

---

Step 7 — Every case, checked

WHAT. Let's make sure no scenario surprises us later. We reuse the parent's worked example: logits $z=[2.0,1.0,0.1]$, true class $t=0$.

Case	Prediction ${\hat{y}}_t$	$\text{CE}=-\log {\hat{y}}_t$	Meaning
Confident & right	${\hat{y}}_t\to 1$	$\to 0$	almost no loss
The worked example	${\hat{y}}_t=0.659$	$0.417$	moderate loss
Pure guessing (2-class)	${\hat{y}}_t=0.5$	$\ln 2=0.693$	learned nothing
Confident & wrong	${\hat{y}}_t\to 0$	$\to +\infty$	savage punishment

WHY cover them all. The whole design of $-\log$ lives in these extremes: it is $0$ when you're perfect, finite in the middle, and unbounded when you're confidently wrong. There is no input that breaks it — except ${\hat{y}}_t=0$ exactly, which is the degenerate case below.

Recall Reveal: verify the worked number

Logits $[2.0,1.0,0.1]$ → $e^{(\cdot )}=[7.389,2.718,1.105]$, sum $=11.212$, so ${\hat{y}}_0=0.659$ and $\text{CE}=-\ln 0.659=0.417$. CE at the worked example ::: $0.417$ CE at pure guessing (${\hat{y}}_t=0.5$) ::: $\ln 2\approx 0.693$

---

The one-picture summary

Everything on this page is one pipeline: logits → softmax → probabilities → compare with one-hot truth via $-\log$ → one loss number → gradient $\hat{y}-y$ flows back.

Recall Feynman: the whole walkthrough in plain words

The model never says "cat" — it pours one litre of belief into three cups (Step 1). The truth is that same litre poured entirely into the right cup (Step 2). To score the model we ask "how surprised are you by the right answer?" and measure surprise with $-\log$, because that function gives zero surprise for certainty, infinite surprise for confident-wrongness, and adds up when facts are independent (Step 3). The scoring formula sums over all cups, but truth is $1$ in only one cup, so all other terms die and we're left with the surprise of the true cup (Step 4). Those cup-heights themselves came from raw scores squashed by softmax — exponentiate to make them positive, divide to make them sum to one (Step 5). And when we ask which way to push the raw scores, the exploding log and the shrinking softmax cancel perfectly, leaving the simplest possible answer: prediction minus truth (Step 6). We checked every corner — perfect, average, coin-flip, and catastrophic — and found the only landmine is feeding a literal zero into the log, which softmax makes impossible (Step 7).

---

Connections

• Parent: Loss functions - MSE, cross-entropy
• The squashing step: Softmax and Sigmoid
• Where the $-\log$ recipe comes from: Maximum Likelihood Estimation
• Who uses the gradient: Gradient Descent via Backpropagation
• Binary special case in practice: Logistic Regression