Cross-entropy concept: Visual walkthrough — AI-ML deep dive

This page builds cross-entropy from nothing. No probability formula is assumed. We start with the single question "how surprised should I feel?" and walk, one picture at a time, until we reach the loss function that trains almost every classifier.

Two conventions fixed up front, used everywhere below.

Which logarithm? Throughout this page $lo g$ means the natural logarithm — the one written $ln$ , whose base is the number $e \approx 2.718$ . We pick it because it is the log built into calculus and gradient-based training. With the natural log, surprise is measured in a unit called the nat (short for "natural unit of information"). Had we used base- $2$ logarithms instead, the same quantities would come out in bits; the shapes and conclusions are identical, only the number-per-unit changes.
Which outcome space? We assume a finite discrete set of possible outcomes — a fixed short list like ${cat, dog, bird}$ — so that every " $\sum_{x}$ " below is an ordinary finite sum that is guaranteed to make sense. If outcomes were continuous (a real number that could be anything), the sum would become an integral, $H (p, q) = - \int p (x) lo g q (x) d x$ , and $p, q$ would be probability densities rather than probabilities; every intuition on this page carries over unchanged, we just replace "sum over a list" with "area under a curve."

Everything else is drawn from the two players in the game:

a true distribution — what reality actually does,
a predicted distribution — what our model guesses reality does.

We will earn every symbol before we use it.

Step 1 — Surprise is a number

WHAT. Before any formula, we agree on one idea: some events shock us, some don't. If I tell you "the sun rose this morning," you shrug. If I tell you "it snowed in the desert," you gasp. Surprise is just how big that gasp is, turned into a number.

Let a probability be written $q$ — a number between $0$ and $1$ telling how likely the model thought an event was. We are going to build a machine that takes in this $q$ and hands back a surprise-number. Let us give that machine a name: write $I (q)$ for "the surprise you feel when an event the model rated $q$ actually happens." Right now $I$ is just a blank box — we have not decided its formula yet, only its name. This step is about deciding what rules the box must obey.

WHY. We want to measure how good a prediction is. The only honest measure is: when the truth is revealed, how shocked were we? A great predictor is rarely shocked; a bad one is shocked constantly.

We demand three things of the blank box $I (q)$ :

If the model was certain it would happen ( $q = 1$ ), surprise must be zero — no shock.
If the model thought it was impossible ( $q \to 0$ ) but it happened, surprise must be enormous.
Surprises from two independent events must add: seeing two independent shocks is one shock plus the other.

PICTURE. In the figure below, the horizontal axis is the model's probability $q$ (running from $0$ on the left to $1$ on the right) and the vertical axis is the surprise $I (q)$ in nats. The curve is high on the left (rare event = shocking), slides down to touch zero on the right (certain event = no shock) — exactly the shape our three demands force.

Intuition Why these three demands leave only the logarithm

Focus on the third demand: for independent events, probabilities multiply ( $q_{1} q_{2}$ ) but surprises must add. So we need a box that converts multiplying-inside into adding-outside: $I (q_{1} q_{2}) = I (q_{1}) + I (q_{2})$ . Try $q = a \cdot b$ built up from many equal pieces: if $I (q^{n}) = n I (q)$ for every whole number $n$ (which the adding rule forces), then the surprise of a probability grows in proportion to how many times you multiplied — and "how many times you multiplied to reach $q$ " is exactly what a logarithm counts. One hidden hypothesis must be stated out loud: we require $I$ to be continuous (equivalently, monotonic) in $q$ . Without it, weird "pathological" solutions exist that satisfy the adding rule only at rational points and jump around chaotically elsewhere. Once we insist $I$ vary smoothly, the adding rule "multiply-in becomes add-out" forces $I (q) = c lo g q$ for some constant $c$ ; the first two demands then fix the sign and scale. That is the sense in which the shape is not chosen but forced.

Step 2 — The logarithm is the only surprise that adds

WHAT. We now fill in the blank box from Step 1 with its forced formula. The surprise of an event the model gave probability $q$ is:

$surprise box I (q) = flip sign so it’s positive - natural log: turns \times into + lo g model’s probability q$

WHY the logarithm and not, say, $1 - q$ ? Because of the additivity demand (together with the continuity assumption stated in Step 1). Independent probabilities multiply: two independent events with probabilities $q_{1}$ and $q_{2}$ happen together with probability $q_{1} q_{2}$ . We wanted their surprises to add. The only continuous function turning " $\times$ inside" into " $+$ outside" is the logarithm:

$lo g (q_{1} q_{2}) = lo g q_{1} + lo g q_{2} \Rightarrow I (q_{1} q_{2}) = I (q_{1}) + I (q_{2}) .$

A rule like $1 - q$ fails this instantly ( $1 - q_{1} q_{2} \neq = (1 - q_{1}) + (1 - q_{2})$ ), so it cannot be surprise.

WHY the minus sign? A probability is at most $1$ , and $lo g$ of a number $\leq 1$ is $\leq 0$ . Surprise should be positive, so we flip the sign with $-$ . The figure shows $- lo g q$ climbing to infinity as $q \to 0$ and touching zero exactly at $q = 1$ — the two anchors we demanded in Step 1.

PICTURE.

Step 3 — Average surprise over the true distribution

WHAT. One event gives one surprise. But reality throws many events at us over time. We want the average surprise per event across a whole day of forecasts.

Now the second player enters. Reality has its own probabilities: write $p (x)$ for how often outcome $x$ actually happens, where $x$ ranges over our fixed finite list of outcomes. Averaging means: weight each outcome's surprise by how often reality serves it up.

$H (p, q) = \sum_{x} how often x truly happens p (x) \times surprise we feel when it happens (- lo g q (x))$

WHY weight by $p$ and not $q$ ? Because reality — not our model — decides how frequently each outcome shows up. If red truly appears 70% of the time, then 70% of our surprises are red-surprises. We feel the surprise our model $q$ assigns, but we feel it as often as reality $p$ dictates. That split — feel $q$ , weighted by $p$ — is the entire heart of cross-entropy.

PICTURE. The bar chart shows outcomes on the x-axis. Each bar's height is the true frequency $p (x)$ ; each bar's color intensity is the surprise $- lo g q (x)$ . Cross-entropy is the total shaded area — frequency times surprise, summed.

Step 4 — The best possible score: entropy

WHAT. What is the lowest score you could ever get? It happens when your model is perfect: $q (x) = p (x)$ everywhere. Then cross-entropy becomes surprise measured against the true frequencies:

$H (p) = - \sum_{x} true freq p (x) lo g same true freq p (x)$

This is the Shannon Entropy of reality — the irreducible surprise built into the world, again in nats.

WHY does a floor exist? Even a perfect forecaster is surprised sometimes, because reality is genuinely random. If a fair coin is 50/50, no forecast removes the shock of not knowing the next flip. That leftover shock is $H (p)$ — the best score physics allows.

PICTURE. Two curves over "how good is the model": cross-entropy $H (p, q)$ is a valley whose lowest point sits exactly at the horizontal line $H (p)$ , reached only when $q = p$ .

Step 5 — The gap above the floor is KL divergence

WHAT. Cross-entropy is the floor plus a penalty for being wrong. Subtract the floor to isolate the penalty:

$wasted surprise D_{K L} (p ∥ q) = your score H (p, q) - best possible H (p) = - \sum_{x} p (x) lo g \frac{q ( x )}{p ( x )}$

Rearranged, the star equation:

$H (p, q) = H (p) + D_{K L} (p ∥ q)$

$H (p)$ — fixed; reality's own randomness. Nothing you do changes it.
$D_{K L} (p ∥ q)$ — the KL Divergence; the extra surprise you pay purely for guessing wrong. Zero only when $q = p$ , positive otherwise.

WHY the two forms agree. Writing the divergence with the ratio $\frac{q ( x )}{p ( x )}$ inside and a leading minus is the same as $+ \sum_{x} p (x) lo g \frac{p ( x )}{q ( x )}$ , because flipping a fraction inside a log flips the sign: $- lo g \frac{q}{p} = lo g \frac{p}{q}$ . Splitting that log, $lo g \frac{p}{q} = lo g p - lo g q$ , and summing gives exactly $H (p, q) - H (p)$ — which is how the box equation is born.

WHY this matters for training. When you train a model you can only move $q$ . Since $H (p)$ is a constant, minimizing $H (p, q)$ is exactly minimizing $D_{K L} (p ∥ q)$ — dragging your guess toward the truth. This is also why minimizing cross-entropy equals Maximum Likelihood Estimation.

PICTURE. A stacked bar: a fixed grey block $H (p)$ on the bottom, a shrinking colored block $D_{K L}$ on top. Training melts away the top block; the bottom never moves.

Step 6 — The degenerate case: one-hot labels

WHAT. In classification, reality is usually certain: the image is a cat. Then $p$ is one-hot — a single $1$ at the true class, $0$ everywhere else, e.g. $p = [1, 0, 0]$ .

Plug it into the sum. Every term with $p (x) = 0$ dies (zero times anything is zero). Only the true class survives:

$H (p, q) = - \sum_{x} p (x) lo g q (x) = - 1 \cdot lo g q (true) - 0 - 0 = surprise at the correct class - lo g q_{true}$

WHY it collapses. With a one-hot truth, only one outcome ever happens, so we only ever feel the surprise attached to that one outcome. The whole sum reduces to the single log of the probability our model gave the right answer.

WHY the entropy floor vanishes too. For a one-hot $p$ , $H (p) = - 1 lo g 1 = 0$ : certain reality has zero built-in randomness. So here $H (p, q) = D_{K L} (p ∥ q)$ exactly — cross-entropy is the KL divergence.

PICTURE. The bar chart from Step 3, but now all bars except one have height $0$ . Only the true-class bar contributes; its shaded area is the whole loss.

Step 7 — Edge cases: zeros, certainty, and the asymmetry

WHAT. Three corners must be shown so you never hit an unshown scenario.

Model says impossible, but it happens ( $q_{true} \to 0$ ): $- lo g q_{true} \to + \infty$ . Loss is unbounded. This is why real code clamps $q$ away from $0$ (e.g. adds a tiny $ε$ ).
A truly impossible outcome ( $p (x) = 0$ ): its term $0 \cdot lo g q (x) = 0$ , dropping out — even if $q (x)$ is tiny. Reality never serves it, so we never feel its surprise. (We define $0 lo g 0 = 0$ by the limit.)
Asymmetry ( $H (p, q) \neq = H (q, p)$ ): swapping the players asks a different question. $H (p, q)$ weights surprise by reality; $H (q, p)$ weights by the model. Different weights, different number.

WHY show these. A learner who only saw the tidy case will divide by zero in code or wrongly swap $p$ and $q$ . The infinity in corner 1 is not a bug — it is cross-entropy screaming "you ruled out the truth."

PICTURE. Left panel: the $- lo g q$ wall rocketing to infinity as $q \to 0$ . Right panel: two side-by-side bars showing $H (p, q)$ and $H (q, p)$ have different heights for the same pair.

The one-picture summary

Everything on one canvas: reality's frequencies $p$ (bar heights) meet the model's surprise $- lo g q$ (curve). Their weighted total is $H (p, q)$ , which splits cleanly into the immovable floor $H (p)$ plus the shrinkable gap $D_{K L} (p ∥ q)$ that training drives to zero.

Recall Feynman retelling of the whole walkthrough

Picture a bag of colored balls. Reality decides the mix — say 70% red, 30% blue; that mix is $p$ . Before each draw you shout your bet: "80% red!"; that bet is $q$ .

When a ball appears, your shock is $- lo g$ (natural log) of the chance you gave that color. Guess a color's chance near zero and it shows up? You gasp huge. Guess it near one and it shows up? You barely blink. We use $lo g$ because two separate shocks should just add up, and only $lo g$ turns "chances multiply" into "shocks add."

Play all day and average your shock, but weigh each color by how often the bag (reality $p$ ) actually gives it. That weighted-average shock is cross-entropy $H (p, q)$ , and its unit is the nat.

The lowest score you could ever hit is when your bets exactly match the bag — but even then you're surprised a little, because the bag is genuinely random. That leftover shock is the entropy $H (p)$ , a floor you can't dig under. Whatever surprise you pay above that floor is pure waste from betting wrong — that's the KL divergence. Since the floor never moves, "get better at guessing" literally means "melt the KL waste to zero," which is exactly what training a classifier does.

Quick self-check

Below, " $lo g$ " is the natural log; work each one by hand so the numbers are never magic.

Coin: reality

p = 0.7

heads, model

q = 0.8

heads. Compute

H (p, q)

- [0.7 lo g 0.8 + 0.3 lo g 0.2] = - [0.7 (- 0.223) + 0.3 (- 1.609)] = - [- 0.156 - 0.483] = 0.639

nats.

Its entropy floor

H (p)

- [0.7 lo g 0.7 + 0.3 lo g 0.3] = - [0.7 (- 0.357) + 0.3 (- 1.204)] = - [- 0.250 - 0.361] = 0.611

nats.

The wasted bits

D_{K L}

H (p, q) - H (p) = 0.639 - 0.611 = 0.028

nats.

Why does one-hot

p

collapse the sum?

Every

p (x) = 0

term dies, leaving only

- lo g q_{true}

Why is

lo g

(not

1 - q

) the surprise rule?

Only

lo g

makes independent surprises add.