1.3.18 · D5Probability & Statistics
Question bank — Entropy and KL divergence
This is a rapid-fire misconception hunter for Entropy and KL divergence. Each line is a Question ::: Answer reveal — read the question, commit to an answer OUT LOUD, then reveal. If you cannot justify the answer in a sentence, you don't own the concept yet. No heavy arithmetic here (that lives in D3/D4) — this page targets the ideas and the edges.
True or false — justify
Entropy can be negative for a discrete distribution.
False. Every term is because makes ; summing non-negatives gives .
The maximum possible entropy of a variable with outcomes is bits.
True. By Gibbs' inequality the uniform distribution maximises entropy, giving ; any bias lowers it.
Adding a new outcome that has probability exactly changes the entropy.
False. By the convention that outcome contributes nothing, so entropy is unchanged.
for all distributions.
False. KL is asymmetric; the two directions ask different questions ("cost of using when truth is " vs. the reverse) and generally give different numbers.
can be negative if is a very bad model.
False. Gibbs'/Jensen's inequality forces always; a worse makes it larger, never negative.
Cross-entropy is always at least as large as the entropy .
True. Since and , cross-entropy can never dip below the optimal code length .
A deterministic variable ( on one outcome) has entropy .
True. There is no surprise: , so you gain zero information when you observe it.
If then and must be identical (where ).
True. Equality in Gibbs' inequality holds iff for every outcome with positive probability.
Entropy measured in nats and in bits describe different amounts of uncertainty.
False. They measure the same uncertainty in different units; nats bits , just a change of log base.
Spot the error
"Because KL measures difference, ."
Wrong — KL is not a metric and violates the triangle inequality; use Jensen-Shannon Divergence if you need a symmetric, triangle-respecting quantity.
", so if some the entropy is undefined."
Wrong — the limit patches this cleanly; a never-occurring event adds zero surprise.
"To minimise classification loss we minimise entropy of the labels."
Wrong — we minimise cross-entropy over the model ; is fixed by the data and cannot be changed. See Cross-Entropy Loss.
"Minimising cross-entropy is unrelated to minimising KL divergence."
Wrong — and is constant in , so minimising one minimises the other exactly.
"If but , KL is just a large finite number."
Wrong — the term diverges, so ; must "cover" every outcome can produce.
"Entropy is a convex function of the distribution ."
Wrong — entropy is concave in ; averaging two distributions never decreases entropy. (KL, by contrast, is convex in the pair .)
"Cross-entropy of a one-hot label uses all class probabilities."
Wrong — with one-hot only the true class survives, so ; the other predictions drop out.
Why questions
Why use in the surprisal instead of, say, ?
The log makes information from independent events add: , matching our intuition that two independent surprises stack, which fails to do.
Why does the uniform distribution — not some peaked one — maximise entropy?
Uniform spreads probability as evenly as possible, so no outcome is predictable; maximum unpredictability means maximum average surprise. This is the seed of the Maximum Entropy Principle.
Why is interpreted as "extra bits wasted"?
It equals cross-entropy minus entropy, — the difference between coding with 's (wrong) code and the optimal -code, i.e. the inefficiency of being wrong.
Why does the direction of KL matter in practice (e.g. in Variational Autoencoders)?
(reverse) is "mode-seeking" — it lets ignore regions where is large, whereas (forward) is "mean-covering" and punishes missing any mass of . Different directions shape the fitted differently.
Why can KL be used as a loss for optimisation even though it is not a distance?
We only need a non-negative quantity that is iff the distributions match and shrinks as they approach — KL does all that; the triangle inequality is irrelevant for gradient descent.
Why does adding more possible outcomes tend to raise the maximum achievable entropy?
The ceiling grows with the number of outcomes ; more distinguishable possibilities means more potential uncertainty to resolve.
Edge cases
What is when one outcome has probability ?
Exactly bits — a certain variable carries no information, the degenerate lower bound of entropy.
What is when exactly?
Exactly — every ratio and ; there is no wasted coding cost.
What happens to if assigns zero probability to an outcome that can produce?
It becomes — an infinitely bad model that rules out something that actually occurs.
What is the entropy contribution of an outcome with ?
Zero, by the convention; it simply doesn't participate in the sum.
As a coin's bias moves from toward or , what happens to its entropy?
It falls monotonically from the maximum bit toward — more predictability, less surprise; the curve is symmetric and peaks at .
What is if the model assigns probability to the true class in a classification task?
It blows up to (from ) — this is why practical implementations clip or smooth predictions away from exact .
For a continuous variable, can "differential entropy" go negative?
Yes — differential entropy replaces the sum with an integral over a density and can be negative (e.g. a narrow Gaussian), unlike discrete entropy which is always .
Recall Quick self-test
Which KL direction gives "mode-seeking" behaviour? ::: The reverse form , used in variational inference. Is entropy concave or convex in ? ::: Concave. When is ? ::: When some outcome has but .
Related: Mutual Information · Information Gain · Evidence Lower Bound (ELBO) · F-divergences · Cross-Entropy Loss