Exercises — Loss functions - MSE, cross-entropy

(a) MSE — the target is a real number; we assume Gaussian noise around the prediction. (b) Cross-Entropy (binary) — the label is a coin flip, so a Bernoulli model fits. (c) Cross-Entropy (categorical) — one of $K=3$ classes, so a Categorical model fits. The rule: real numbers ⇒ Gaussian ⇒ MSE; categories ⇒ Bernoulli/Categorical ⇒ CE.

• $y$ is the true label, exactly $0$ or $1$.
• $\hat{y}$ is the model's predicted probability that $y=1$.
• $\hat{y}$ must be in the open interval $(0,1)$ — never exactly $0$ or $1$, because $\log 0=-\infty$. This is why we pass raw scores through a sigmoid first.

WHAT: subtract, square, average. Errors $y-\hat{y}=[2-3,4-4,6-4]=[-1,0,2]$. Squares $=[1,0,4]$. Why square? To make every error positive and to punish the big miss (the $2$) far more than a small one. $\text{MSE}=\frac{1+0+4}{3}=\frac{5}{3}\approx 1.667$

Since $y=1$, the $(1-y)$ term vanishes: $\text{BCE}=-[1\cdot \log 0.8+0]=-\log 0.8\approx 0.223$ A confident-and-correct guess ⇒ small loss. Good.

Step 1 — exponentiate (softmax numerator turns any real number into a positive weight): $e^1=2.71828,e^2=7.38906,e^0=1$. Step 2 — normalize (divide by the sum so weights become probabilities that add to 1): Sum $=11.10734$. Probs $=[0.2447,0.6652,0.0900]$. Step 3 — pick the true class (one-hot kills the other two terms): $\text{CE}=-\log (0.6652)\approx 0.4076$

If $y=1$: $\text{BCE}=-\log 0.5$. If $y=0$: $\text{BCE}=-[0+\log (1-0.5)]=-\log 0.5$. Either way: $\text{BCE}=-\log 0.5=\log 2\approx 0.693$ Meaning: $0.693$ is the "learned nothing" line. If training loss sits near $\ln 2$, the model is no better than a coin flip. It is your first sanity-check baseline.

Confident and right: $\hat{y}=0.99$ ⇒ $\text{BCE}=-\log 0.99\approx 0.01005$. Confident and wrong: the model is 99% sure of class 0, i.e. $\hat{y}=P(y=1)=0.01$ ⇒ $\text{BCE}=-\log 0.01\approx 4.60517$. Look at figure s01: near $\hat{y}=1$ the curve hugs the floor, but as $\hat{y}\to 0$ it rockets to infinity.

Why this shape matters: $-\log$ blows up near $0$, so a confident wrong answer earns a savage penalty ($\approx 458\times$ the confident-right loss here). That steep wall is exactly what forces the network to correct bold mistakes fast — a fairness the flatter MSE curve does not provide.

(a) CE: $\hat{y}-y=0.01-1=-0.99$. A strong push toward the truth. (b) MSE: $(0.01-1)(0.01)(0.99)=(-0.99)(0.0099)=-0.009801$. A tiny push — about $101\times$ weaker. The extra factor $\hat{y}(1-\hat{y})$ shrinks to $\approx 0$ exactly when the sigmoid saturates (near $0$ or $1$). So MSE-on-sigmoid stalls precisely when the model is boldly wrong — the worst time to stall. See Backpropagation for how this factor enters the chain rule, and Gradient Descent for why a vanishing gradient freezes learning.

WHAT: write the two-class sum explicitly. Let class 1 be "positive" with probability ${\hat{y}}_1=\hat{y}$, and class 0 be "negative" with probability ${\hat{y}}_0=1-\hat{y}$ (they must sum to 1). Let the one-hot truth be $y_1=y$, $y_0=1-y$. $\text{CE}=-[y_1\log {\hat{y}}_1+y_0\log {\hat{y}}_0]=-[y\log \hat{y}+(1-y)\log (1-\hat{y})].$ WHY it matters: BCE is not a separate law — it is categorical CE with the redundant second probability folded in via $1-\hat{y}$. One idea, two dresses.

• The first term $\frac{1}{2}\log (2\pi {\sigma }^2)$ contains no $\hat{y}$ — it is a constant as far as our knob is concerned, so it cannot move the location of the minimum. Drop it.
• The second term is $\frac{1}{2{\sigma }^2}(y-\hat{y}{)}^2$. The factor $\frac{1}{2{\sigma }^2}$ is a positive constant multiplier; scaling a function by a positive number does not move where its minimum sits.
• What remains that actually depends on $\hat{y}$ is $(y-\hat{y}{)}^2$. Conclusion: $\arg {\min }_{\hat{y}}(-\log p)=\arg {\min }_{\hat{y}}(y-\hat{y}{)}^2$. Averaging over $n$ points gives MSE. This is why MSE is Gaussian maximum likelihood in disguise.

$\Vert w{\Vert }^2=3^2+4^2=9+16=25$. Penalty $=\frac{0.1}{2}\cdot 25=0.05\cdot 25=1.25$. $\text{Total}=0.40+1.25=\mathrm{1.65.}$ Why add it: the data-loss alone can be minimized by wild, large weights that overfit; the penalty makes big weights cost something, trading a little training fit for better generalization.

Step 1 — softmax derivative. Differentiating the ratio gives the standard identity $\frac{\partial {\hat{y}}_k}{\partial z_i}={\hat{y}}_k({\delta }_{ki}-{\hat{y}}_i),$ where ${\delta }_{ki}=1$ if $k=i$ else $0$. (This is just the quotient rule applied to $e^{z_k}/{\sum }_j{e}^{z_j}$.) Step 2 — chain rule into CE. $\frac{\partial \text{CE}}{\partial z_i}=-{\sum }_k\frac{y_k}{{\hat{y}}_k}\cdot \frac{\partial {\hat{y}}_k}{\partial z_i}=-{\sum }_k\frac{y_k}{{\hat{y}}_k}{\hat{y}}_k({\delta }_{ki}-{\hat{y}}_i)=-{\sum }_k{y}_k({\delta }_{ki}-{\hat{y}}_i).$ Step 3 — split the sum. $=-{\sum }_k{y}_k{\delta }_{ki}+{\hat{y}}_i{\sum }_k{y}_k=-y_i+{\hat{y}}_i\cdot 1={\hat{y}}_i-y_i,$ using ${\sum }_k{y}_k=1$ (one-hot sums to one) and ${\sum }_k{y}_k{\delta }_{ki}=y_i$. WHY it matters: the messy softmax Jacobian collapses to prediction − truth, the same beautiful form MSE-with-identity-output has. Clean gradients, no vanishing when confidently wrong.

$\text{BCE}(0.9)=-\log 0.9\approx 0.10536$. $\text{BCE}(0.1)=-\log 0.1\approx 2.30259$. Ratio $=\frac{2.30259}{0.10536}\approx 21.85>20$. ✓ The loss doesn't grow linearly as you get worse — it grows ferociously near the wrong end, which is the whole point of $-\log$.

This is a regression target that merely lives in $[0,1]$; it is a continuous quantity, not a Bernoulli outcome. The clean choice is MSE (or a bounded regression loss), because the noise model is roughly Gaussian on the fraction, not a coin flip. The subtlety: BCE is only justified when $y$ is genuinely $0/1$ and $\hat{y}$ is a probability of that event. Using BCE with fractional $y$ (so-called "soft labels") is sometimes done deliberately, but it silently changes the assumed model. Rule of thumb: ask "is my target a bet outcome (use CE) or a measured amount (use MSE)?" — the range alone does not decide it. Compare with Logistic Regression, where the target really is $0/1$ and BCE is exactly right.

You have mastered this page if you can, from a blank sheet:

• compute MSE, BCE, and softmax-CE from raw inputs (L2),
• state and use the coin-flip baseline $\ln 2$ (L3),
• explain the vanishing-gradient stall of MSE-on-sigmoid (L3),
• derive BCE from categorical CE and MSE from the Gaussian (L4),
• prove the softmax+CE gradient equals $\hat{y}-y$ (L5).