1.3.19 · D5Probability & Statistics
Question bank — Cross-entropy concept
This is a rapid-fire self-test for the ideas in Cross-entropy concept (index 1.3.19). Every item is a single reveal line: read the prompt, say your answer out loud with a reason, then check. The goal is not arithmetic (that lives in the computational decks) — it is to catch the sneaky conceptual traps that make people think they understand cross-entropy when they don't.
Before we start, one shared vocabulary reminder so no symbol is unearned:
True or false — justify
TF1. Cross-entropy is symmetric: .
False. The two put the on different distributions, so the weights differ; answers "reality , belief " while asks a totally different question. See KL Divergence for the same asymmetry.
TF2. Cross-entropy can be smaller than the entropy of the truth, .
False. Since and KL is always , cross-entropy can never dip below the Shannon Entropy .
TF3. exactly when .
True. The gap is , which is zero only when the two distributions match everywhere.
TF4. Minimizing cross-entropy over the model is the same as minimizing KL divergence.
True. does not depend on the model, so minimizing moves only the term. The constant is dead weight during training.
TF5. Cross-entropy and Maximum Likelihood Estimation push the parameters in the same direction.
True. Averaging over the data is the negative log-likelihood; minimizing cross-entropy on the empirical distribution = maximizing likelihood.
TF6. For a one-hot label, cross-entropy equals of the probability assigned to the correct class.
True. Every wrong-class term is multiplied by and vanishes, leaving only .
TF7. Cross-entropy is bounded above — there is a worst possible value.
False. If the model assigns to an outcome that actually occurs, . A confident-and-wrong prediction has unbounded loss.
TF8. Swapping to Categorical Cross-Entropy for a 3-class problem changes the underlying formula.
False. Categorical cross-entropy is the same ; the name just signals a one-hot over several classes.
Spot the error
SE1. "Cross-entropy = , always — just take negative log of the prediction."
The error: that formula only holds for hard (one-hot) labels. With soft labels every class contributes , so you must sum over all outcomes.
SE2. "We weight each outcome's surprise by the model's probability ."
The error: you weight by the true probability . Reality decides how often each surprise is felt; the model only sets how big each surprise is.
SE3. "Minimizing cross-entropy minimizes the entropy of the truth."
The error: is fixed by reality and untouchable. You are shrinking the part by moving , not shrinking .
SE4. "Cross-entropy penalizes wrong predictions linearly, like squared error."
The error: the penalty is , which blows up as . A confident wrong guess costs enormously more than a mildly wrong one — that steep gradient is exactly why we prefer it to MSE.
SE5. "Since , and both terms are entropies, cross-entropy is just a bigger entropy."
The error: is a divergence, not an entropy; it measures mismatch between two distributions, and only the first term is a genuine entropy.
SE6. "If the model outputs raw scores, we can plug them straight into ."
The error: must be a valid probability distribution (non-negative, sums to 1). You first pass scores through Softmax Function (or a sigmoid for binary Logistic Regression) to get legal probabilities.
SE7. "Cross-entropy loss can be negative because of the minus sign."
The error: probabilities satisfy , so , and for every term. The total is always non-negative.
SE8. " and are the same number."
The error: KL is asymmetric, mirroring TF1. Which distribution sits in the reference slot changes both the weighting and the answer.
Why questions
WHY1. Why is the surprise of an event defined as and not, say, ?
Because we want independent events to have additive surprise: when . Only turns a product of probabilities into a sum.
WHY2. Why do we weight surprises by when computing average surprise?
Because is the actual long-run frequency of outcome ; averaging "how often × how surprising" gives the true expected cost over many trials.
WHY3. Why does cross-entropy beat mean squared error for classification?
Its gradient is inversely proportional to the predicted probability, so confidently-wrong predictions get a huge corrective push; MSE's gradient shrinks near saturated outputs, stalling learning.
WHY4. Why must the model output pass through softmax before cross-entropy?
Cross-entropy needs a valid probability distribution (, ). Softmax Function converts arbitrary scores into exactly that.
WHY5. Why can we drop during optimization?
It depends only on the fixed ground-truth distribution, not on model parameters, so its gradient with respect to the model is zero.
WHY6. Why does Label Smoothing turn a one-hot label into a soft distribution?
It replaces the perfect with something like , so the "vanishing" wrong-class terms reappear, discouraging the model from becoming infinitely confident (which would demand infinite logits).
WHY7. Why is cross-entropy called a "surprise" measure and not a "distance"?
Because it isn't symmetric and doesn't satisfy the triangle inequality; it measures the coding cost of using the wrong code, which behaves like expected surprise, not geometric distance.
WHY8. Why does the term for a class with contribute nothing, even though might be huge?
The product ; an outcome that never happens contributes no expected surprise regardless of how the model rates it.
WHY9. Why is minimizing cross-entropy connected to reducing Mutual Information-style mismatch between model and data?
Both are built from information terms; cross-entropy specifically drives toward , aligning the model's information content with the data's true structure.
Edge cases
EC1. What is cross-entropy when the model is perfect, ?
It equals , the Shannon Entropy of the truth — the irreducible floor. You cannot beat it because reality itself carries that much uncertainty.
EC2. What happens to if the model assigns probability exactly to the true class?
It diverges to . This is why implementations clamp probabilities (e.g. add a tiny ) — an unhedged, wrong, 100%-confident prediction is infinitely costly.
EC3. If the truth is deterministic (), what is , and what does cross-entropy reduce to?
(no uncertainty in reality), so cross-entropy equals outright.
EC4. What is the cross-entropy of a uniform prediction over classes, for any one-hot label?
— the same for every sample. This is the baseline "know nothing" loss a fresh model starts near.
EC5. Two classes, true , model : what is ?
nats, which equals because the model matches truth, so .
EC6. Can cross-entropy be zero? Under what condition?
Only if the truth is deterministic () and the model is perfectly certain of the correct outcome (). Any residual uncertainty in forces .
EC7. What happens to gradients as the model becomes correct-and-confident ()?
The loss and its gradient shrinks toward zero, so training naturally slows once predictions are already right — no wasted correction.
EC8. In multi-label classification (an image is both "cat" and "outdoor"), is standard softmax cross-entropy appropriate?
No — softmax forces the probabilities to compete and sum to 1, but multiple labels can be true simultaneously. You use per-label binary cross-entropy (independent sigmoids) instead.
Recall One-line self-check
If you can state why and play different roles in , you have defused most of these traps at once. What role does play vs ? ::: (true distribution) sets how often each surprise occurs; (model) sets how large each surprise is.