Exercises — Log-loss - binary cross-entropy

WHAT we do: plug $y=1$ into $\ell =-[y\log \hat{y}+(1-y)\log (1-\hat{y})]$. WHY: $y=1$ makes the coefficient $(1-y)=0$, so the second term dies. Only $-\log \hat{y}$ remains — this is the parent's "$-\log (\text{prob of the true class})$" in action. $\ell =-[1\cdot \log 0.7+0\cdot \log 0.3]=-\log 0.7\approx \mathrm{0.357.}$ Small-ish loss because the model leaned the right way (0.7 > 0.5) but wasn't fully confident.

WHAT: now $y=0$, so the first term dies and $-\log (1-\hat{y})$ survives. WHY: the surviving factor is the probability the model gave to the actual class, which is class 0, whose probability is $1-\hat{y}=0.3$. $\ell =-\log (1-0.7)=-\log 0.3\approx \mathrm{1.204.}$ Bigger than Exercise 1.1: the model bet against the truth here.

WHAT: whichever term survives, the surviving probability is $0.5$. WHY: if $y=1$ we get $-\log 0.5$; if $y=0$ we get $-\log (1-0.5)=-\log 0.5$. Same either way. $\ell =-\log 0.5=\log 2\approx \mathrm{0.693.}$ This is the baseline the parent note names: any useful model must score below $0.693$ nats on average.

WHAT: compute each ${\ell }_i$ (only the label-matching term), then average. ${\ell }_1=-\log 0.9=0.10536,{\ell }_2=-\log (1-0.2)=-\log 0.8=0.22314,{\ell }_3=-\log 0.4=\mathrm{0.91629.}$ WHY average: $J=\frac{1}{N}{\sum }_i{\ell }_i$ makes the number independent of batch size $N$ (parent formula). $J=\frac{1}{3}(0.10536+0.22314+0.91629)=\frac{1}{3}(1.24479)\approx \mathrm{0.4149.}$ The third point (right class but only 0.4 confidence, i.e. leaning wrong) dominates the loss.

WHAT: use the parent's cancellation result $\frac{\partial \ell }{\partial z}=\hat{y}-y$. WHY this tool: we differentiate w.r.t. $z$ (not $\hat{y}$) because $z=w^{\top }x+b$ is what gradient descent actually pushes. The messy $\hat{y}(1-\hat{y})$ from the Sigmoid function cancels, leaving the clean error term. $\frac{\partial \ell }{\partial z}=\hat{y}-y=0.4-1=-\mathrm{0.6.}$ Interpret: negative gradient on $z$ means gradient descent will increase $z$ (step is $z\leftarrow z-\eta \cdot (\text{grad})$, and subtracting a negative adds), pushing $\hat{y}$ up toward $1$ — correct, since the true label is 1.

WHAT: apply $\frac{\partial \ell }{\partial w}=(\hat{y}-y)x$ and $\frac{\partial \ell }{\partial b}=(\hat{y}-y)$. WHY: chain rule multiplies the $z$-gradient $(\hat{y}-y)$ by $\partial z/\partial w=x$ and by $\partial z/\partial b=1$. $\hat{y}-y=0.75-0=\mathrm{0.75.}$ $\frac{\partial \ell }{\partial w}=0.75\times 2=1.5,\frac{\partial \ell }{\partial b}=\mathrm{0.75.}$ Both positive → descent decreases $w$ and $b$, lowering $z$ and pulling $\hat{y}$ down toward $0$. Correct, since $y=0$.

WHAT: the surviving term is $-\log \hat{y}$. As $\hat{y}\to 0^{+}$, $\log \hat{y}\to -\infty$, so $\ell =-\log \hat{y}\to +\infty .$ WHY it matters: infinite penalty for being certain and wrong — the model can never "give up" on the truth. See figure below (red curve blows up on the left). Contrast MSE: $(\hat{y}-y{)}^2=(0-1{)}^2=1$ in the limit. MSE caps the punishment at a finite $1$, so a confidently-wrong point produces only a mild signal. Combined with the vanishing ${\sigma }^{\prime }$ factor from the parent note, that's why MSE + sigmoid stalls. See Mean Squared Error.

WHAT: substitute $y^{\prime }=1-y$ and ${\hat{y}}^{\prime }=1-\hat{y}$ into $\ell$. ${\ell }^{\prime }=-[(1-y)\log (1-\hat{y})+(1-(1-y))\log (1-(1-\hat{y}))]$ $=-[(1-y)\log (1-\hat{y})+y\log \hat{y}]=\ell .$ WHY intuition matches: relabelling "class 1" as "class 0" (and swapping the predicted probability to match) is just renaming the coin's faces. The loss shouldn't care what we call the classes — only how much probability landed on the true one.

WHAT / values:

• $\hat{y}=1$: $\ell =-\log 1=0$ (perfect, unreachable in practice).
• $\hat{y}=0.5$: $\ell =-\log 0.5=0.693$ (baseline).
• $\hat{y}\to 0$: $\ell \to +\infty$. Trend: strictly decreasing as $\hat{y}$ climbs toward the truth $1$. Convexity: $\frac{d^2}{d{\hat{y}}^2}(-\log \hat{y})=\frac{1}{{\hat{y}}^2}>0$ for all $\hat{y}\in (0,1)$, so yes, convex in $\hat{y}$. (Convexity in the weights $w$ additionally relies on $z=w^{\top }x+b$ being linear — see the Logistic Regression note.) The blue curve in the figure above shows this bowl-shaped-on-a-log-axis decay.

(a) WHAT / likelihood: independence ⇒ multiply single-point Bernoulli probabilities (parent's Maximum Likelihood Estimation chain): $L={\hat{y}}_1^{y_1}(1-{\hat{y}}_1{)}^{1-y_1}\cdot {\hat{y}}_2^{y_2}(1-{\hat{y}}_2{)}^{1-y_2}=0.6^1\cdot 0.4^0\cdot 0.1^0\cdot 0.9^1.$ $L=0.6\times 1\times 1\times 0.9=\mathrm{0.54.}$ (b) WHAT / negative-log-average: $-\frac{1}{2}\log L=-\frac{1}{2}\log 0.54=-\frac{1}{2}(-0.61619)=\mathrm{0.30809.}$ Confirm via mean log-loss: ${\ell }_1=-\log 0.6=0.51083,{\ell }_2=-\log 0.9=0.10536,$ $J=\frac{1}{2}(0.51083+0.10536)=\mathrm{0.30809.}✓$ They match because $-\frac{1}{N}\log \prod =\frac{1}{N}\sum (-\log )$ — the log turns the product into the sum, exactly the parent derivation.

WHAT: take true label $y=1$, so $p=(p_0,p_1)=(0,1)$, and $q=(1-\hat{y},\hat{y})$. $H(p,q)=-[p_0\log q_0+p_1\log q_1]=-[0\cdot \log (1-\hat{y})+1\cdot \log \hat{y}]=-\log \hat{y}.$ For $y=0$, $p=(1,0)$ gives $H=-\log (1-\hat{y})$. WHY this is exactly $\ell$: both cases collapse to $-\log (\text{prob of true class})$ — identical to the single-example log-loss. This is precisely why the loss is called binary cross-entropy; see Cross-Entropy and KL Divergence.

WHAT: divide numerator and denominator by $e^{z_1}$: $q_1=\frac{e^{z_1}}{e^{z_0}+e^{z_1}}=\frac{1}{e^{z_0-z_1}+1}=\frac{1}{1+e^{-(z_1-z_0)}}=\sigma (z_1-z_0).$ WHY it matters: the softmax over 2 classes depends only on the logit difference $z_1-z_0$, which behaves exactly like a single sigmoid input. Categorical cross-entropy then reduces to the binary log-loss we've been using — binary is the special case of Softmax and Categorical Cross-Entropy.

(a) Forward pass. With $w=b=0$, $z_i=0$ for both, so ${\hat{y}}_1=\sigma (0)=0.5,{\hat{y}}_2=\sigma (0)=\mathrm{0.5.}$ (A fresh model is maximally unsure — the baseline state.)

(b) Loss. Both surviving terms are $-\log 0.5$: $J=\frac{1}{2}[-\log 0.5+(-\log 0.5)]=-\log 0.5=\mathrm{0.693.}$ Exactly the always-0.5 baseline — expected, since the model is outputting 0.5.

(c) Gradients. Error terms: ${\hat{y}}_1-y_1=0.5-1=-0.5$; ${\hat{y}}_2-y_2=0.5-0=0.5$. $\frac{\partial J}{\partial w}=\frac{1}{2}{\sum }_i({\hat{y}}_i-y_i)x_i=\frac{1}{2}[(-0.5)(1)+(0.5)(-2)]=\frac{1}{2}(-0.5-1.0)=-\mathrm{0.75.}$ $\frac{\partial J}{\partial b}=\frac{1}{2}{\sum }_i({\hat{y}}_i-y_i)=\frac{1}{2}(-0.5+0.5)=0.$

(d) Update. $\theta \leftarrow \theta -\eta \nabla$: $w\leftarrow 0-0.5\times (-0.75)=0.375,b\leftarrow 0-0.5\times 0=0.$

(e) Should the loss drop? Yes. The step raised $w$ to $0.375$. Now $z_1=0.375\cdot 1=0.375>0$ so ${\hat{y}}_1>0.5$ (pushed toward its target $1$), and $z_2=0.375\cdot (-2)=-0.75<0$ so ${\hat{y}}_2<0.5$ (pushed toward its target $0$). Both predictions moved the right way, and because log-loss is convex here, this step decreases $J$. The figure traces this move.

(a) WHAT breaks: $-\log 1.0=0$ is fine here, but the danger is the mirror case — $y=1$, $\hat{y}=0.0$ gives $-\log 0=+\infty \to$ NaN once it enters a sum. Even at $\hat{y}=1$, if the other label were involved you'd hit $\log (1-1)=\log 0$. Any exact 0 or 1 poisons the batch with NaN, which then propagates through every gradient (parent's second [!mistake]). (b) Clipping. Replace $\hat{y}$ with $\min (\max (\hat{y},\epsilon ),1-\epsilon )$. For $\hat{y}=1.0$, clip to $1-10^{-7}$: $\ell =-\log (1-10^{-7})\approx 1.0000000\times 10^{-7}\approx 10^{-7}.$ A tiny finite loss instead of a $0/\text{NaN}$ landmine. (For $y=1$, $\hat{y}=0$ clipped to $\epsilon$: $\ell =-\log (10^{-7})\approx 16.118$ — huge but finite, exactly the harsh-but-survivable punishment we want.)