Loss functions - MSE, cross-entropy: Question bank

Quick symbol reminder so line one is followable:

$y$ = the truth (the correct answer we wanted).
$\overset{y}{^}$ = the network's prediction (its guess).
$L$ = the loss, one number saying "how wrong was $\overset{y}{^}$ ".
$n$ = the number of training examples we average the loss over.
$σ$ (sigma) = the spread (standard deviation) of the assumed Gaussian noise in regression — it measures how noisy the targets are.
$z$ = a logit, the raw pre-activation score before softmax/sigmoid turns it into a probability.
$δ_{k m}$ = the Kronecker delta: it equals $1$ when the two indices match ( $k = m$ ) and $0$ when they differ. Think of it as an "is-this-the-same-slot?" switch.
$ε$ (epsilon) = a tiny positive number (like $1 0^{- 7}$ ) used as a safety margin to keep probabilities away from the exact $0$ and $1$ where $lo g$ explodes.
MSE = Mean Squared Error (regression). CE = Cross-Entropy (classification); BCE = Binary Cross-Entropy, which is simply CE in the special case of exactly $K = 2$ classes.
NLL = Negative Log-Likelihood — the single recipe both losses come from.

The one recipe, shown once (so the rest of the page stands on it)

Before the traps, let us actually watch the recipe both losses obey, so nothing below is asserted blindly.

Figure — Loss functions - MSE, cross-entropy

Formula Why softmax+CE and sigmoid+CE both give gradient

$\overset{y}{^} - y$ (the cancellation, shown) Sigmoid case. Let $\overset{y}{^} = σ (z) = \frac{1}{1 + e ^{- z}}$ . A classic fact: $\frac{d y ^}{d z} = \overset{y}{^} (1 - \overset{y}{^})$ — this is the "saturating factor" that dies when $\overset{y}{^}$ is near $0$ or $1$ . BCE is $L = - [y lo g \overset{y}{^} + (1 - y) lo g (1 - \overset{y}{^})]$ , so $\frac{\partial L}{\partial y ^} = - \frac{y}{y ^} + \frac{1 - y}{1 - y ^} = \frac{y ^ - y}{y ^ ( 1 - y ^ )} .$ Chain rule multiplies by $\frac{d y ^}{d z} = \overset{y}{^} (1 - \overset{y}{^})$ , and the factor cancels exactly: $\frac{\partial L}{\partial z} = \frac{y ^ - y}{y ^ ( 1 - y ^ )} \cdot \overset{y}{^} (1 - \overset{y}{^}) = \overset{y}{^} - y .$ Softmax case. With $\overset{y}{^}_{k} = \frac{e ^{z_{k}}}{\sum _{j} e ^{z_{j}}}$ and one-hot $y$ , differentiating $L = - \sum_{k} y_{k} lo g \overset{y}{^}_{k}$ through softmax uses $\frac{\partial y ^ _{k}}{\partial z _{m}} = \overset{y}{^}_{k} (δ_{k m} - \overset{y}{^}_{m})$ , where $δ_{k m}$ (the Kronecker delta defined in the symbol list) is $1$ only when $k = m$ ; summing the terms telescopes to the same clean result $\frac{\partial L}{\partial z _{m}} = \overset{y}{^}_{m} - y_{m}$ . Look at the figure: MSE-on-sigmoid (black) flattens to near-zero slope when confidently wrong, but CE-on-sigmoid (red) keeps a strong slope there.

True or false — justify

Both MSE and cross-entropy are just different names for the same formula.

False. They are the same recipe (negative log-likelihood) applied to different noise models — Gaussian for MSE, Bernoulli/Categorical for cross-entropy — so the resulting formulas differ.

Minimizing loss and maximizing likelihood are opposite goals.

False. They are the same goal. We take

- lo g

of the likelihood (log is monotonic, so it preserves the location of the best

θ

) and flip the sign, turning "maximize probability" into "minimize loss".

The constant

\frac{1}{2} lo g (2 π σ^{2})

in MSE's derivation must be kept or the answer is wrong.

False. Here

σ

is the fixed noise spread, so this term does not depend on

\overset{y}{^}

; it shifts the loss by a constant and never moves the minimum. We drop it for cleanliness; the optimal weights are unchanged.

Cross-entropy can be negative.

False. It is

- lo g (a probability in (0, 1])

, and

lo g

of a number

\leq 1

\leq 0

, so with the minus sign CE

\geq 0

. It reaches exactly

0

only when the true-class probability is

1

A loss of zero means the network is perfect on training data.

True but dangerous. Zero training loss only means it fits the seen data perfectly; that often signals overfitting, which is why we use Regularization and a held-out set.

MSE always has a smaller numerical value than cross-entropy, so it is "easier" to optimize.

False. The raw magnitude of a loss is meaningless across different losses — they live on different scales. What matters is the shape of the surface and the gradient, not the number's size.

Squaring the error in MSE is done only to remove the sign.

False. It removes the sign and penalizes large errors disproportionately: an error of

2

costs

4 \times

an error of

1

, not

2 \times

, which pushes the model to fix big misses first.

BCE is a different loss from categorical cross-entropy.

False. BCE is exactly categorical CE specialized to

K = 2

: one sigmoid output

\overset{y}{^} = P (y = 1)

implies

P (y = 0) = 1 - \overset{y}{^}

, and the two-term Categorical sum collapses into the compact BCE formula.

The

\div n

in MSE averages over the

K

output components of a single prediction.

False. Here

n

is the number of examples; the

\div n

averages the per-example squared errors, not the components of one output vector.

Spot the error

"I fed raw logits

z = [2.0, - 1.0]

straight into

- lo g

to get cross-entropy."

The input to CE's log must be a valid probability in

(0, 1)

. Raw logits can be negative, so

lo g

of a negative is undefined (NaN). Fix: apply softmax/sigmoid first, or use a combined logits-to-loss function.

"For my cat-vs-dog classifier I used MSE because outputs are just numbers."

With a sigmoid output, MSE's gradient carries a

\overset{y}{^} (1 - \overset{y}{^})

factor that vanishes when the model is confidently wrong, so learning stalls. Cross-entropy cancels that factor (see the cancellation shown above) and keeps the gradient

\overset{y}{^} - y

"CE

= - \sum_{k} y_{k} lo g \overset{y}{^}_{k}

, so I summed the log-terms over all

K

classes including the wrong ones."

Not an error to write the full sum, but note

y_{k} = 0

for every wrong class, so those terms are

0 \cdot lo g (\cdot) = 0

. Only the true-class term

- lo g \overset{y}{^}_{true}

actually contributes.

"My BCE gradient is huge for a confident correct answer, which seems wasteful."

For a correct confident answer (

\overset{y}{^} \to y

) the gradient

\overset{y}{^} - y \to 0

, so it is small, not huge. The huge gradients happen for confident wrong answers — exactly where you want a big correction.

"I normalized my softmax probabilities to sum to 1, then re-applied softmax to be safe."

Softmax already outputs a valid probability distribution summing to

1

. Re-applying it distorts the values (squashes them toward uniform) and breaks the clean softmax+CE gradient

\overset{y}{^}_{k} - y_{k}

"I averaged CE over classes (

\div K

) the same way MSE averages over samples (

\div n

)."

Wrong axis: the

\div n

averages over examples, not components. CE for one example already collapses to a single term

- lo g \overset{y}{^}_{true}

; you average over examples, never over the class index.

Why questions

Why does the softmax+CE gradient simplify to the same

\overset{y}{^} - y

as MSE-with-identity-output?

Because differentiating

- lo g \overset{y}{^}_{true}

through softmax uses

\frac{\partial y ^ _{k}}{\partial z _{m}} = \overset{y}{^}_{k} (δ_{k m} - \overset{y}{^}_{m})

, and the terms telescope to

\overset{y}{^}_{m} - y_{m}

(shown in the derivation above). It is the same prediction-minus-truth signal MSE gives with an identity output, which feeds clean gradients into Backpropagation.

Why does

- lo g

blow up as the predicted probability approaches

0

p \to 0^{+}

lo g p \to - \infty

, so

- lo g p \to + \infty

. This encodes "infinite surprise" at seeing an outcome you called impossible, punishing confident wrong answers savagely.

Why does the choice of loss depend on the task rather than the network architecture?

The loss encodes an assumption about how the data's noise/labels are distributed (Gaussian for real values, Categorical for classes). That assumption comes from the nature of the target, which is a property of the task, not the layers.

Why is cross-entropy preferred over MSE specifically when a sigmoid or softmax sits at the output?

Because

\frac{d σ ( z )}{d z} = \overset{y}{^} (1 - \overset{y}{^})

is a factor that vanishes when

\overset{y}{^}

is near

0

1

; MSE keeps that factor and its gradient dies there, while CE's

lo g

produces a matching

\frac{1}{y ^ ( 1 - y ^ )}

that cancels it, leaving the live gradient

\overset{y}{^} - y

— see Logistic Regression and the cancellation figure above.

Why do we take the log of the likelihood at all, instead of maximizing the raw product?

The product of many tiny probabilities underflows to

0

in floating point, and products are hard to differentiate.

lo g

turns the product into a sum (easy gradients) and is monotonic, so it does not move the optimum.

Why is a BCE hovering near

0.693

a red flag during training?

0.693 = ln 2

is the loss of a model outputting

\overset{y}{^} = 0.5

(pure coin-flip) on every example. Staying there means the model has learned nothing discriminative.

Why does MSE-on-probabilities give a "poorly calibrated" classifier?

MSE assumes Gaussian noise, which is false for discrete

{0, 1}

labels, so its optimum does not match the true probability. The output numbers then fail to mean "actual chance of the class".

Edge cases

What is the BCE if the model predicts exactly

\overset{y}{^} = 1

but the truth is

y = 0

It is

- lo g (1 - \overset{y}{^}) = - lo g (0) = + \infty

. In practice we clip

\overset{y}{^}

into

[ε, 1 - ε]

(with

ε

a tiny positive number like

1 0^{- 7}

) to avoid infinities and NaNs.

What happens to MSE's loss surface if all targets

y

are identical constants?

MSE still has a unique minimum where

\overset{y}{^}

equals that constant. As a function of a single scalar output it is a parabola; in the full weight space it is a paraboloid (a multidimensional bowl) with one lowest point — nothing degenerates.

For a multi-class problem with

K

classes, what CE does a totally uninformed model (uniform

\overset{y}{^}_{k} = 1/ K

) achieve?

- lo g (1/ K) = lo g K

, the maximum-uncertainty baseline. For

K = 2

this is

lo g 2 = 0.693

, matching the binary coin-flip case.

If a true class has predicted probability

\overset{y}{^}_{true} = 1

exactly, what is CE and is that fine?

= - lo g 1 = 0

, the perfect score. It is mathematically fine, but reaching it usually needs infinite logits, so real training only approaches it — and forcing it invites overfitting.

What does MSE's gradient

\overset{y}{^} - y

do at the exact minimum where

\overset{y}{^} = y

It is

0

, so Gradient Descent takes no step — the model has stopped being wrong on that point and correctly leaves it alone.

In binary classification with a single sigmoid output, why can we use the compact BCE instead of the general

K

-class CE?

A single probability

\overset{y}{^} = P (y = 1)

implies

P (y = 0) = 1 - \overset{y}{^}

, so the two-class Categorical collapses to the Bernoulli form

- [y lo g \overset{y}{^} + (1 - y) lo g (1 - \overset{y}{^})]

. They are the same thing for

K = 2

What if two classes tie with equal top probability in softmax — is the loss undefined?

No. CE only reads the true class's probability

\overset{y}{^}_{true}

; ties among other classes never appear in the formula, so the loss is perfectly well-defined.

Recall One-line summary to lock in

Every trap here reduces to two facts: (1) each loss is an NLL for a specific noise model (derived, not asserted, above), and (2) both clean gradients are $\overset{y}{^} - y$ because the activation's derivative cancels. If an answer contradicts either, it is the misconception.