Cross-entropy concept

Overview

Cross-entropy measures the difference between two probability distributions by quantifying how many bits (on average) we need to encode samples from one distribution if we wrongly assume another distribution. It's the workhorse loss function for classification in machine learning because it directly penalizes confident wrong predictions.

Why it matters: Cross-entropy bridges information theory and machine learning. It tells us "how surprised" we are when reality doesn't match our model's beliefs.

[!intuition] The Core Idea

Imagine you're a weather forecaster. Every day you predict rain probability, and nature reveals the truth.

• If you say 90% rain and it rains → small surprise (you were confident and correct)
• If you say 10% rain and it rains → huge surprise (you were confident and wrong)

Cross-entropy is the average surprise your predictions cause, weighted by what actually happens. The better your predictions match reality, the lower the cross-entropy.

Key insight: Cross-entropy = "How many extra bits am I wasting by using the wrong probability distribution?"

[!definition] Mathematical Definition

Given:

• $p(x)$ = true probability distribution (reality, ground truth)
• $q(x)$ = predicted probability distribution (model's belief)

The cross-entropy from $p$ to $q$ is:

$H(p, q) = -\sum_{x} p(x) \log q(x)$

For continuous distributions:

$H(p, q) = -\int p(x) \log q(x) \, dx$

Units: bits (if $\log = \log_2$) or nats (if $\log = \ln$). In ML we use natural log.

[!formula] Derivation from First Principles

Step 1: Shannon Entropy (Self-Information)

Question: How surprised am I when event $x$ with probability $p(x)$ occurs?

Answer: The surprise (self-information) is:

$I(x) = -\log p(x)$

Why this formula?

• Rare events ($p(x)$ small) → large surprise
• Certain events ($p(x) = 1$) → zero surprise
• Independent events should have additive surprise: $I(x, y) = I(x) + I(y)$, which means we need $\log$

Expected surprise when sampling from $p(x)$:

$H(p) = -\sum_{x} p(x) \log p(x)$

This is the Shannon entropy of $p$ — the minimum average bits needed to encode samples from $p$ if we know $p$ perfectly.

Step 2: What If We Use the Wrong Code?

Suppose reality follows $p(x)$ but we design our encoding assuming distribution $q(x)$.

• We assign code length $-\log q(x)$ to event $x$ (optimal for $q$)
• But events occur with frequency $p(x)$
• Average code length = $-\sum_{x} p(x) \log q(x)$

This is cross-entropy $H(p, q)$ — the average bits we actually use with the wrong distribution.

Why this step? Cross-entropy tells us the cost of being wrong. If $q = p$, we use the minimum possible bits. Any deviation from $p$ wastes bits.

Step 3: Connection to KL Divergence

The extra bits wasted by using $q$ instead of $p$:

$D_{KL}(p \parallel q) = H(p, q) - H(p) = -\sum_{x} p(x) \log \frac{q(x)}{p(x)}$

This is the Kullback-Leibler divergence.

Key relationship:

$H(p, q) = H(p) + D_{KL}(p \parallel q)$

• $H(p)$ = irreducible uncertainty (entropy of truth)
• $D_{KL}(p \parallel q)$ = extra cost from model mismatch

Why minimize cross-entropy in ML? Since $H(p)$ is fixed (we can't change reality), minimizing $H(p, q)$ is equivalent to minimizing $D_{KL}(p \parallel q)$ — making our model match reality.

[!example] Example 1: Binary Classification (Coin Flip)

Setup: True distribution: coin lands heads with $p = 0.7$. Model predicts $q = 0.8$.

Truth: $p(\text{heads}) = 0.7$, $p(\text{tails}) = 0.3$

Model: $q(\text{heads}) = 0.8$, $q(\text{tails}) = 0.2$

Cross-entropy:

$H(p, q) = -[0.7 \log(0.8) + 0.3 \log(0.2)]$

Why this step? We weight each outcome by its true probability $p$, not the model's belief $q$. Reality determines how often each surprise occurs.

$H(p, q) = -[0.7 \times (-0.223) + 0.3 \times (-1.609)]$ $= -[-0.156 - 0.483] = 0.639 \text{ nats}$

Compare to entropy of truth:

$H(p) = -[0.7 \log(0.7) + 0.3 \log(0.3)] = 0.611 \text{ nats}$

Wasted bits: $D_{KL} = 0.639 - 0.611 = 0.028$ nats.

Interpretation: Our model is slightly overconfident in heads, causing small information loss.

[!example] Example 2: Multiclass Classification (3 classes)

Setup: Image classification — cat, dog, bird.

Truth (one-hot): $p = [1, 0, 0]$ (it's definitely a cat)

Model predicts: $q = [0.7, 0.2, 0.1]$

Cross-entropy:

$H(p, q) = -[1 \times \log(0.7) + 0 \times \log(0.2) + 0 \times \log(0.1)]$

Why zero out the other terms? Because $p(\text{dog}) = p(\text{bird}) = 0$ — these classes don't occur in this sample. We only care about the probability assigned to what actually happened.

$H(p, q) = -\log(0.7) = 0.357 \text{ nats}$

What if model predicted $q = [0.1, 0.8, 0.1]$ (wrong class)?

$H(p, q) = -\log(0.1) = 2.303 \text{ nats}$

Why the huge jump? Cross-entropy heavily penalizes confident wrong predictions. Model was 80% sure it was a dog, but it was a cat — massive surprise.

[!example] Example 3: Why Not Mean Squared Error?

Question: Why use cross-entropy for classification instead of MSE?

Setup: Binary classification, true label $y = 1$ (positive class).

Model outputs: $\hat{y} = 0.1$ (predicts negative).

MSE: $L_{MSE} = (1 - 0.1)^2 = 0.81$

Cross-entropy: $L_{CE} = -\log(0.1) = 2.303$

Why this difference matters:

If model improves to $\hat{y} = 0.2$:

• MSE: $(1 - 0.2)^2 = 0.64$ → decrease of $0.17$
• Cross-entropy: $-\log(0.2) = 1.609$ → decrease of $0.694$

Why cross-entropy is better: It gives stronger gradient signal when the model is confidently wrong. MSE gradient is linear in error; cross-entropy gradient is inversely proportional to the predicted probability, so wrong predictions get corrected faster.

[!mistake] Common Pitfalls

Mistake 1: "Cross-entropy is just negative log probability"

Why it feels right: In practice, we write loss = -log(p_correct_class), which looks like we're ignoring the sum.

Why it's incomplete: That's only true when the true distribution is one-hot (hard labels). The full definition sums over all outcomes weighted by $p(x)$.

The fix:

• One-hot $p$ → $H(p,q) = -\log q(y_{\text{true}})$ (all other terms vanish)
• Soft labels $p$ → $H(p,q) = -\sum_i p_i \log q_i$ (all terms matter)

When it matters: Label smoothing, knowledge distillation, multi-label classification.

Mistake 2: "Minimizing cross-entropy = minimizing entropy"

Why it feels right: Both have "entropy" in the name and use $-\log$.

Why it's wrong:

• $H(p)$ = entropy of true distribution (fixed, doesn't depend on model)
• $H(p, q)$ = cross-entropy (depends on model's $q$)

The fix: You minimize $H(p, q)$ by adjusting $q$ to match $p$. Since $H(p, q) = H(p) + D_{KL}(p \parallel q)$ and $H(p)$ is constant, you're really minimizing $D_{KL}(p \parallel q)$.

Mistake 3: "Cross-entropy is symmetric"

Why it feels right: Both $p$ and $q$ appear in the formula.

Why it's wrong: $H(p, q) \neq H(q, p)$ in general.

Proof: $H(p, q) = -\sum p(x) \log q(x)$ $H(q, p) = -\sum q(x) \log p(x)$

Different weights, different sums.

The fix: Cross-entropy measures "surprise when reality is $p$ but we assume $q$". Swapping them asks a different question. Always keep $p$ as ground truth, $q$ as model.

[!recall]- Feynman Explanation (ELI12)

Imagine you're playing a guessing game. Every round, I secretly pick a colored ball from a bag (maybe 70% red, 30% blue). You have to guess the color before I show you.

Now, you get points based on how confident you were:

• If you say "90% sure it's red" and it IS red → you get lots of points (you were right and confident!)
• If you say "90% sure it's red" and it's BLUE → you lose TONS of points (you were wrong and super confident)

Cross-entropy is like keeping track of your average point loss over many rounds. If your guesses match the true bag contents (70% red), you lose very few points. If you always guess 50-50 when the bag is actually 70-30, you lose more points because you're surprised more often.

In machine learning, the model is you guessing, and reality is the bag. We train the model to minimize cross-entropy = minimize average surprise = make better guesses that match reality!

[!mnemonic] Memory Aid

CROSS = Compare Reality's Occurrence to System's Speculation

• Compare: Two distributions
• Reality: True $p(x)$ (ground truth)
• Occurrence: Weighted by true probabilities
• System: Model's predicted $q(x)$
• Surprise: $-\log q(x)$ (information content)

Formula mnemonic: "People Love Quality" → $p(x) \log q(x)$ summed with a negative sign.

Connections

• Shannon Entropy — cross-entropy generalizes self-entropy when $p \neq q$
• KL Divergence — $H(p,q) = H(p) + D_{KL}(p \parallel q)$
• Softmax Function — converts logits to probabilities $q$ for cross-entropy loss
• Logistic Regression — binary cross-entropy is its loss function
• Categorical Cross-Entropy — multiclass extension used in neural networks
• Label Smoothing — regularization technique that modifies $p$ to prevent overconfidence
• Maximum Likelihood Estimation — minimizing cross-entropy = maximizing likelihood
• Mutual Information — related measure of shared information between variables

Flashcards

Concept Map

Hinglish (regional understanding)