Log-loss - binary cross-entropy

---

WHY do we even need a new loss? (Steel-man MSE first)

For linear regression we used Mean Squared Error. It's tempting to reuse it: $(\hat y - y)^2$.

---

WHAT is the loss? — derived from first principles

HOW we turn probability into a loss (derivation):

1. We want the parameters that make the observed data most likely (Maximum Likelihood).
2. Likelihood of $N$ independent points: $L = \prod_{i=1}^N \hat y_i^{\,y_i}(1-\hat y_i)^{1-y_i}$. Why product? Independence ⇒ joint probability multiplies.
3. Products are painful; take $\log$ (monotone, so the maximizer is unchanged): $\log L = \sum_i \big[\, y_i\log\hat y_i + (1-y_i)\log(1-\hat y_i)\,\big]$ Why log? Turns product into sum, and avoids underflow of tiny probabilities.
4. Maximizing log-likelihood = minimizing its negative. Average over $N$ to make it scale-independent:

---

---

The gradient — where the magic cancellation happens

Let $z = w^\top x + b$ and $\hat y = \sigma(z) = \dfrac{1}{1+e^{-z}}$. Key sigmoid fact: $\sigma'(z) = \sigma(z)\big(1-\sigma(z)\big) = \hat y(1-\hat y)$

Take the single-example loss $\ell = -[y\log\hat y + (1-y)\log(1-\hat y)]$ and differentiate w.r.t. $z$:

= \frac{\hat y - y}{\hat y(1-\hat y)}$$ *Why this step?* Just chain rule on each log term; combine over a common denominator. $$\frac{\partial \ell}{\partial z} = \frac{\partial \ell}{\partial \hat y}\cdot\frac{d\hat y}{dz} = \frac{\hat y - y}{\hat y(1-\hat y)}\cdot \hat y(1-\hat y) = \boxed{\hat y - y}$$ > [!intuition] Why this is beautiful > The messy $\hat y(1-\hat y)$ **cancels**. The gradient is just **prediction − truth**, exactly like linear regression. So: > $$\frac{\partial J}{\partial w_j} = \frac{1}{N}\sum_i (\hat y_i - y_i)\,x_{ij}, \qquad \frac{\partial J}{\partial b} = \frac{1}{N}\sum_i(\hat y_i - y_i)$$ > No vanishing gradient when confidently wrong — the error term $(\hat y - y)$ stays large. --- ## Worked examples > [!example] Confident and correct > True $y=1$, model says $\hat y = 0.99$. > $\ell = -[1\cdot\log 0.99 + 0] = -\log 0.99 \approx 0.01$. > **Why this step?** $y=1$ kills the $(1-y)$ term. Tiny loss = good. > [!example] Confident and WRONG (the punishment) > True $y=1$, model says $\hat y = 0.01$. > $\ell = -\log 0.01 = \log 100 \approx 4.6$. > **Why this step?** Same surviving term, but now $\log$ of a tiny number is a big negative → big positive loss. As $\hat y\to 0$, $\ell\to\infty$: never be sure and wrong. > [!example] Full-batch computation > Two points: $(y=1,\hat y=0.8)$, $(y=0,\hat y=0.3)$. > $\ell_1 = -\log 0.8 = 0.223$. $\ell_2 = -\log(1-0.3)=-\log 0.7 = 0.357$. > $J = \tfrac12(0.223+0.357) = 0.29$. > **Why this step?** Average the per-example losses; each uses only the term matching its label. --- ## Forecast-then-Verify > [!recall]- Forecast: what is the log-loss of a model that always predicts $\hat y = 0.5$? > Every point contributes $-\log 0.5 = \log 2 \approx 0.693$, regardless of label. So $J = \ln 2 \approx 0.693$. This is the **baseline** — any useful model must beat $0.693$ nats (or $1$ bit if using $\log_2$). --- > [!mistake] "Log-loss can be negative." > **Why it feels right:** it has a minus sign out front. > **The fix:** $\hat y\in(0,1)\Rightarrow \log\hat y < 0$, and the leading minus flips it positive. Log-loss is always $\ge 0$, hitting 0 only in the (unreachable) limit of perfect confidence on the true class. > [!mistake] "Feed exactly 0 or 1 into the loss." > **Why it feels right:** a perfect prediction *is* 0 or 1. > **The fix:** $\log 0 = -\infty$ → NaN. In practice clip $\hat y$ to $[\varepsilon, 1-\varepsilon]$ (e.g. $\varepsilon=10^{-7}$), or better, use a numerically stable `logits`-based implementation. --- ## Feynman > [!recall]- Explain to a 12-year-old > Imagine you bet points on whether it will rain. If you say "90% sure it rains" and it rains, you barely lose anything. If you scream "I'm 99% SURE it won't rain!" and it pours, you lose a *ton* of points. Log-loss is that point system: being humble costs little, being loud-and-wrong costs a lot. The computer adjusts its guesses to lose as few points as possible. > [!mnemonic] **"Log the Right one."** > Log-loss = **−log(probability you gave the RIGHT answer)**. Right answer got high prob → small loss. Right answer got tiny prob → huge loss. --- ## Active-recall flashcards #flashcards/ai-ml Why not use MSE with a sigmoid output? ::: MSE + sigmoid is non-convex and its gradient has a $\sigma'=\hat y(1-\hat y)$ factor that vanishes when confidently wrong, stalling learning. From what principle is log-loss derived? ::: Maximum likelihood of a Bernoulli distribution (then take negative log and average). Write the Bernoulli likelihood of one label. ::: $P(y\mid\hat y)=\hat y^{y}(1-\hat y)^{1-y}$. State single-example log-loss. ::: $\ell=-[y\log\hat y+(1-y)\log(1-\hat y)]$. What does log-loss equal in words? ::: Minus the log of the probability the model assigned to the true class. What is $\partial\ell/\partial z$ for sigmoid + log-loss? ::: $\hat y - y$ (the $\hat y(1-\hat y)$ cancels). Gradient of $J$ w.r.t. $w_j$? ::: $\frac{1}{N}\sum_i(\hat y_i-y_i)x_{ij}$. Log-loss of an always-0.5 model? ::: $\ln 2\approx 0.693$ (baseline to beat). Can log-loss be negative? ::: No; $\ge 0$ always, since $-\log(\text{prob in }(0,1))\ge0$. Why clip $\hat y$? ::: To avoid $\log 0=-\infty$ / NaN; clip to $[\varepsilon,1-\varepsilon]$. --- ## Connections - [[Sigmoid function]] — supplies $\hat y=\sigma(z)$ and the $\sigma'=\hat y(1-\hat y)$ identity. - [[Logistic Regression]] — model whose loss this is. - [[Maximum Likelihood Estimation]] — the origin of the whole formula. - [[Cross-Entropy and KL Divergence]] — log-loss is cross-entropy between the true label distribution and the predicted one. - [[Gradient Descent]] — consumes the clean $(\hat y - y)x$ gradient. - [[Mean Squared Error]] — the loss we rejected here, and why. - [[Softmax and Categorical Cross-Entropy]] — the multi-class generalization. ## 🖼️ Concept Map ```mermaid flowchart TD LR[Logistic regression outputs prob y-hat] MSE[MSE with sigmoid] NC[Non-convex + vanishing gradient] MLE[Maximum Likelihood] BER[Bernoulli model] LIK[Likelihood = product] LOGL[Log-likelihood = sum] BCE[Binary Cross-Entropy loss] SINGLE[-log prob of true class] SIG[Sigmoid derivative y-hat 1-y-hat] GRAD[Gradient y-hat - y] LR -->|tempting reuse| MSE MSE -->|suffers from| NC NC -->|fix: switch to| MLE MLE -->|assumes| BER BER -->|independence gives| LIK LIK -->|take log| LOGL LOGL -->|negate + average| BCE BCE -->|one term survives| SINGLE SIG -->|cancels in| GRAD BCE -->|differentiate| GRAD GRAD -->|clean update, no stall| LR ``` ## 🔊 Hinglish (regional understanding) > [!intuition]- Hinglish mein samjho > Dekho, logistic regression ka output ek **probability** $\hat y$ hota hai — matlab "mujhe kitna yakeen hai ki label 1 hai". Ab hume ek loss chahiye jo confident-and-correct hone par chhota ho, aur confident-and-wrong hone par bahut bada ho. Yahi kaam log-loss (binary cross-entropy) karta hai: $\ell = -[y\log\hat y + (1-y)\log(1-\hat y)]$. Simple words mein — ye hai **minus log of the probability tumne sahi answer ko di**. Sahi class ko high probability di to loss chhota, aur galti se sahi class ko tiny probability di to $\log$ bada negative ho jaata hai, loss aasman chhoo leta hai. > > Ye formula aasman se nahi aaya — hum **Bernoulli distribution ki maximum likelihood** se derive karte hain. Har point ek coin flip hai probability $\hat y$ ke saath. Saare points ka product lo, phir $\log$ lo (product ko sum bana deta hai aur chhoti probabilities ko underflow se bachata hai), negative karo, aur average lo — bas, log-loss taiyaar. > > MSE kyun nahi use karte? Kyunki sigmoid ke saath MSE **non-convex** ho jaata hai aur uska gradient tab mar jaata hai jab model confidently galat hota hai — matlab jab sabse zyada correction chahiye, tab hi learning ruk jaati hai. Log-loss ka gradient super clean nikalta hai: $\hat y - y$ (wahi $\hat y(1-\hat y)$ cancel ho jaata hai). Bilkul linear regression jaisa — prediction minus truth, guna feature. Isiliye classification mein log-loss standard hai. Bas dhyaan rakhna: $\hat y$ ko exactly 0 ya 1 mat feed karna, warna $\log 0 = -\infty$ NaN de dega; usse $[\varepsilon, 1-\varepsilon]$ mein clip kar do. ![[audio/2.2.10-Log-loss---binary-cross-entropy.mp3]]