Log-loss - binary cross-entropy: Question bank

Before you start, make sure these three symbols feel automatic:

$y \in {0, 1}$ — the true label, a hard 0 or 1.
$\overset{y}{^} \in (0, 1)$ — the model's predicted probability that the label is 1 (the open interval means it can get close to 0 or 1 but never touch them).
$ℓ = - [y lo g \overset{y}{^} + (1 - y) lo g (1 - \overset{y}{^})]$ — the loss for one example. Because $y$ is 0 or 1, only one bracket term ever survives.

True or false — justify

Log-loss can take negative values.

False.

\overset{y}{^} \in (0, 1)

lo g \overset{y}{^} < 0

; the leading minus flips it to a positive number, and the floor is

0

(only reached in the unreachable limit of perfect confidence on the true class).

Minimising log-loss is the same as maximising the Bernoulli likelihood of the data.

True. Log is monotone increasing, so it preserves the location of the maximum; negating and averaging turns "maximise likelihood" into "minimise mean negative log-likelihood", which is exactly log-loss.

For a single example, both the

y lo g \overset{y}{^}

term and the

(1 - y) lo g (1 - \overset{y}{^})

term contribute to the loss.

False. Since

y

is exactly 0 or 1, one term is multiplied by

0

and vanishes; the loss is always just

- lo g (prob assigned to the true class)

A model that always outputs

\overset{y}{^} = 0.5

has zero log-loss because it never commits.

False. Each point costs

- lo g 0.5 = ln 2 \approx 0.693

; that is the baseline every useful model must beat, not zero.

If two models have the same classification accuracy, they must have the same log-loss.

False. Accuracy only cares which side of

0.5

you land on; log-loss also punishes how confidently you are right or wrong, so a timid-but-correct model and a bold-but-correct model score differently.

Log-loss and cross-entropy are different losses that happen to look similar.

False. Binary log-loss is the cross-entropy between the one-hot true distribution

{y, 1 - y}

and the predicted

{\overset{y}{^}, 1 - \overset{y}{^}}

; it is the same quantity written for the two-class case.

Swapping

lo g

base from

e

2

changes which parameters minimise the loss.

False. Changing base multiplies the whole loss by a constant (

lo g_{2} x = ln x / ln 2

), and scaling by a positive constant never moves the minimiser — only the numeric loss value changes (nats vs bits).

The gradient

\partial ℓ / \partial z = \overset{y}{^} - y

is a lucky coincidence special to this one example.

False. It holds for every example and stems from a deliberate cancellation between

\partial ℓ / \partial \overset{y}{^}

and the sigmoid derivative

\overset{y}{^} (1 - \overset{y}{^})

; that clean form is precisely why log-loss was chosen.

Spot the error

"Log-loss uses squared error inside because we still want to penalise big mistakes."

Error: there is no square anywhere. The penalty comes from

- lo g

, which grows unboundedly as the true class's probability drops toward 0 — far harsher than a square, which stays finite.

"With the sigmoid, MSE is fine because

\overset{y}{^} - y

is still the error we shrink."

Error: with

\overset{y}{^} = σ (z)

, MSE is non-convex in the weights and its gradient carries an extra

σ^{'} (z) = \overset{y}{^} (1 - \overset{y}{^})

factor that vanishes when the model is confidently wrong, so learning stalls exactly when it shouldn't.

"To be safe against tiny probabilities, feed

\overset{y}{^} = 0

when the model is sure the label is 0."

Error:

lo g 0 = - \infty

, so if the true label were actually 1 you'd get infinite/NaN loss. Always clip

\overset{y}{^}

into

[ε, 1 - ε]

instead of using exact 0 or 1.

"Since only one log term survives, the

(1 - y) lo g (1 - \overset{y}{^})

half is useless and can be deleted."

Error: it survives whenever

y = 0

. Both halves are needed so the single formula covers both label values; deleting one breaks every negative example.

"Log-loss is derived by choosing a formula that happens to be convex."

Error: convexity was not assumed — it falls out of maximum-likelihood on a Bernoulli model. We derived from a probability principle and got convexity as a bonus.

"The

\frac{1}{N}

in front of

J

changes the optimal weights, so drop it for speed."

Error: dividing by the constant

N

rescales the loss and its gradient uniformly, leaving the minimiser unchanged; it only keeps the number scale-independent across batch sizes.

"Because the gradient is

\overset{y}{^} - y

, log-loss and MSE on a sigmoid give identical updates."

Error: MSE's gradient is

(\overset{y}{^} - y) \overset{y}{^} (1 - \overset{y}{^}) x

— it carries the extra shrinking factor. Log-loss's gradient is the clean

(\overset{y}{^} - y) x

with no such factor, which is the whole point.

Why questions

Why take the logarithm of the likelihood at all?

It turns the product over independent points into a sum (easier to differentiate) and prevents numerical underflow from multiplying many tiny probabilities; being monotone, it doesn't move the optimum.

Why does confidently wrong cost so much more than mildly wrong?

The loss is

- lo g

of the true class's probability; as that probability shrinks toward 0,

- lo g

shoots to

+ \infty

, so screaming a wrong answer is punished far more steeply than hedging.

Why does the messy

\overset{y}{^} (1 - \overset{y}{^})

cancel in the gradient?

The chain rule multiplies

\partial ℓ / \partial \overset{y}{^} = (\overset{y}{^} - y) / [\overset{y}{^} (1 - \overset{y}{^})]

by the sigmoid derivative

\overset{y}{^} (1 - \overset{y}{^})

; the denominator and the sigmoid factor are identical and divide out, leaving

\overset{y}{^} - y

Why do we average over

N

instead of summing?

Averaging makes the loss (and its gradient magnitude) independent of how many points are in the batch, so a learning rate tuned on one batch size still behaves sensibly on another.

Why is a Bernoulli — not a Gaussian — the right noise model here?

The target

y

is a binary outcome (0 or 1), which a Bernoulli describes exactly; a Gaussian assumes continuous, symmetric noise and would lead you back to MSE, which we rejected.

Why must

\overset{y}{^}

live in the open interval

(0, 1)

rather than

[0, 1]

The sigmoid asymptotes to 0 and 1 but never reaches them, and

lo g

of the endpoints is

- \infty

; staying strictly inside keeps the loss finite and its gradient defined.

Edge cases

What is the loss when the prediction is perfect (

y = 1, \overset{y}{^} \to 1

It tends to

0

but never exactly reaches it, since

\overset{y}{^}

can approach 1 only in the limit; log-loss has an infimum of 0 that is approached, not attained.

What happens to the loss as a confidently-wrong prediction gets worse (

y = 1, \overset{y}{^} \to 0

It diverges to

+ \infty

; the harsh, unbounded penalty on the true class's vanishing probability is the deliberate design feature that discourages overconfident errors.

For a completely uncertain model (

\overset{y}{^} = 0.5

) does the loss depend on the true label?

No — both classes get probability

0.5

, so the loss is

- lo g 0.5 = ln 2

whether

y = 0

y = 1

; the baseline is label-blind.

If a class is perfectly separable, why can log-loss keep decreasing without bound during training?

The model can push

z = w^{⊤} x + b

ever larger, driving

\overset{y}{^}

toward 1 on positives and 0 on negatives; the loss keeps shrinking toward 0, so weights can blow up unless regularisation or early stopping intervenes.

Does log-loss handle an imbalanced dataset (say 99% negatives) on its own?

Not automatically — every point contributes equally, so a model can score low loss by always predicting the majority class; you must add class weights or resampling if the rare class matters.

How does binary log-loss relate to its multi-class cousin?

It is the two-class special case of categorical cross-entropy; replace the sigmoid with a softmax over

K

classes and log-loss generalises to

- lo g (prob of the true class)

summed over one-hot labels.

Connections

Log-loss - binary cross-entropy — the parent note these traps stress-test.
Sigmoid function — its derivative $\overset{y}{^} (1 - \overset{y}{^})$ is what cancels in the gradient.
Mean Squared Error — the rejected alternative several traps compare against.
Maximum Likelihood Estimation — origin of the loss, referenced in the "why" traps.
Cross-Entropy and KL Divergence — the identity behind the "same quantity" trap.
Gradient Descent — consumer of the clean $(\overset{y}{^} - y) x$ gradient.
Logistic Regression — the model whose loss this is.
Softmax and Categorical Cross-Entropy — the multi-class generalisation in the last edge case.