Foundations — Cross-entropy concept

Before you can read a single line of the parent note, you need to own every symbol it throws at you. This page assumes you have seen none of them. We build each one from a picture, in the order that each one leans on the one before it.

0. What is a "probability" and how do we picture it?

A probability is just a number between $0$ and $1$ that says how likely something is.

• $0$ means "never happens"
• $1$ means "always happens"
• $0.7$ means "happens 7 times out of 10 in the long run"

The picture: think of a bar of length $1$ chopped into pieces. Each piece is one possible outcome, and the width of the piece is its probability. Because something must happen, all the pieces together fill the whole bar — the widths add up to $1$.

Why the topic needs it: cross-entropy compares two of these sliced bars — reality's bar and the model's bar.

1. The symbols $x$, $p(x)$, $q(x)$

Now we can name the pieces.

The picture: two bars stacked on top of each other. Same slices ("cat", "dog", "bird"), but the widths differ — because the model doesn't know reality perfectly.

Why the topic needs it: the whole game of cross-entropy is "reality happens with frequency $p(x)$, but our model only believed $q(x)$." Two bars, one truth, one guess.

2. The summation symbol $\sum$

${\sum }_{x}p(x)$

The picture: you point at each slice of the bar in turn, read off its number, and drop it into a running total.

${\sum }_{x}p(x)=p(\text{cat})+p(\text{dog})+p(\text{bird})$

Why the topic needs it: cross-entropy is an average over all outcomes. Sigma is how we say "loop over every slice and total up the surprise, weighted by how often it happens."

3. The logarithm $\log$

This is the one symbol most people fear, so we build it slowly from a picture.

Why this tool and not, say, a square root or a plain fraction? Because we want a function that turns multiplying into adding. That single property — $\log (a\times b)=\log (a)+\log (b)$ — is exactly what we need later: surprises from two independent events should add, but their probabilities multiply. The log is the only tool that converts one into the other.

The picture: for probabilities (numbers between $0$ and $1$), the log curve dives downward. As $p\to 0$ (rarer and rarer), $\log p\to -\infty$. As $p\to 1$ (certain), $\log p\to 0$.

Why the topic needs it: log turns tiny probabilities into big numbers with a minus sign in front — the raw material of "surprise."

4. Surprise: the minus-log $-\log p(x)$

Notice the log of a probability is always negative or zero (the curve lives below the axis). Surprise should be a positive feeling, so we flip the sign.

The picture: take the diving log curve from figure s03 and flip it above the axis. Now it rises toward infinity as $p\to 0$ — "the rarer it is, the more shocked I am."

Sanity check on the cases:

• $p(x)=1$ → $-\log 1=0$. No surprise, you knew it. ✔
• $p(x)=0.5$ → $-\ln 0.5=0.69$ nats. Mild surprise. ✔
• $p(x)=0.01$ → $-\ln 0.01=4.6$ nats. Big surprise. ✔
• $p(x)\to 0$ → surprise $\to \infty$. An "impossible" event happening is infinitely shocking. ✔

Why the topic needs it: cross-entropy is average surprise. This is the thing being averaged. See Shannon Entropy for the case where you average surprise using the same distribution you're surprised by.

5. Averaging surprise: $H(p)$ and $H(p,q)$

Now stitch $\sum$ and surprise together.

Read the shape of $H(p,q)$ left to right:

1. Walk over every slice $x$ (that's the ${\sum }_x$).
2. Ask "how often does this slice really happen?" → weight $p(x)$.
3. Ask "how surprised was the model?" → $-\log q(x)$.
4. Multiply weight × surprise, and total them up.

The result is the model's average surprise across reality, which is the whole topic in one formula.

6. The gap between them: $D_{KL}$

The picture: two stacked bars — total height $H(p,q)$ (model's surprise) sits on top of the shorter $H(p)$ (reality's unavoidable surprise). The extra sliver on top is $D_{KL}$.

Why the topic needs it: $H(p)$ is fixed (you can't change reality). So training a model to shrink $H(p,q)$ is the same as shrinking that sliver $D_{KL}$ — pushing your guess bar to match reality's bar. See KL Divergence.

7. Where the guesses come from: one-hot labels & softmax

Two last symbols the parent leans on.

Why the topic needs it: in classification, the true bar is one-hot and the guessed bar is a softmax output. Cross-entropy between them is the loss you minimise (see Categorical Cross-Entropy and Logistic Regression).

The prerequisite map

Read it top to bottom: probabilities give distributions; logs give surprise; sigma averages surprise into entropy and cross-entropy; their gap is KL; softmax and one-hot supply the two bars; everything feeds the final loss.

Equipment checklist

Self-test: can you answer each before revealing?