Log-loss - binary cross-entropy: Worked examples

Intuition What this page is for

The parent note gave you the formula and one clean gradient. This page throws every kind of input at it: both labels, confident/unsure predictions, the two dangerous edges ( $\overset{y}{^} \to 0$ and $\overset{y}{^} \to 1$ ), a full batch, a real-world story, and an exam-style trap. If you can work all of these, no log-loss question can surprise you.

Recall the one formula we lean on the whole way (from the parent and Logistic Regression): $ℓ = - [y lo g \overset{y}{^} + (1 - y) lo g (1 - \overset{y}{^})]$ $y$ is the true label (either $0$ or $1$ ). $\overset{y}{^} \in (0, 1)$ is the model's predicted probability that the label is $1$ , produced by the Sigmoid function. " $lo g$ " here means natural log $ln$ (base $e$ ), so our units are nats. Here $ℓ$ is the loss of one example; later (Ex 5) we average many such $ℓ$ values into a batch cost we will call $J$ .

The scenario matrix

Every log-loss situation is one cell in this table. The columns are the model's prediction; the rows are the true label. The extra rows below are the special/degenerate/word cases.

Case class	What makes it special	Example #
$y = 1$ , $\overset{y}{^}$ high (correct + confident)	best case, loss near 0	Ex 1
$y = 1$ , $\overset{y}{^}$ low (wrong + confident)	punished, loss large	Ex 2
$y = 0$ , $\overset{y}{^}$ low (correct + confident)	mirror of Ex 1, uses the other term	Ex 3a
$y = 0$ , $\overset{y}{^}$ high (wrong + confident)	mirror of Ex 2	Ex 3b
$\overset{y}{^} = 0.5$ (maximally unsure)	baseline, same loss for either label	Ex 4
Full batch (average of many single losses)	how single losses combine into one number	Ex 5
Degenerate $\overset{y}{^} \to 0$ or $\to 1$	limiting behaviour, NaN danger, clipping	Ex 6
Real-world word problem	translate a story into $y, \overset{y}{^}$	Ex 7
Exam twist: $lo g_{2}$ vs $ln$ , and a gradient	units + the $(\overset{y}{^} - y)$ update	Ex 8

The figure below plots the two "surviving-term" curves so you can see every cell before we compute it.

Figure — Log-loss - binary cross-entropy

Figure s01 — two chalk curves on a dark board. The blue curve $- lo g \overset{y}{^}$ rises from $0$ at the right ( $\overset{y}{^} = 1$ ) to $\infty$ at the left ( $\overset{y}{^} = 0$ ): this is the loss when the truth is $y = 1$ . The pink curve $- lo g (1 - \overset{y}{^})$ is its mirror, rising to $\infty$ at the right: the loss when $y = 0$ . A yellow dashed line marks the $ln 2 = 0.693$ baseline, and the two curves cross there at $\overset{y}{^} = 0.5$ . Labelled dots mark the Ex 1, Ex 2 and Ex 3b heights.

The worked examples

Worked example Ex 3 — Cells 3a & 3b:

$y = 0$ correct AND $y = 0$ wrong (the mirror term) Now the truth is $y = 0$ . Compute for (3a) $\overset{y}{^} = 0.05$ (correct) and (3b) $\overset{y}{^} = 0.95$ (wrong).

Forecast: which term of the formula survives now, and which of (3a)/(3b) should hurt?

With $y = 0$ , the first term $y lo g \overset{y}{^} = 0$ ; the second term $(1 - y) lo g (1 - \overset{y}{^})$ survives. Why this step? $y = 0$ kills the first term. The model's belief in the true class is now $1 - \overset{y}{^}$ .
(3a) $ℓ = - lo g (1 - 0.05) = - lo g (0.95) = 0.0513$ nats. Why this step? Truth is class 0; the model gave class 0 a probability of $1 - \overset{y}{^} = 0.95$ . Great.
(3b) $ℓ = - lo g (1 - 0.95) = - lo g (0.05) = 2.996$ nats. Why this step? The model gave the true class only $0.05$ probability. Confident and wrong.

Verify: (3a) equals Ex 1's answer and (3b) equals Ex 2's — perfect mirror symmetry. On the figure, (3a)/(3b) are read off the pink curve at $\overset{y}{^} = 0.05$ and $\overset{y}{^} = 0.95$ . ✓

Worked example Ex 4 — Cell: maximally unsure,

$\overset{y}{^} = 0.5$ Model refuses to commit: $\overset{y}{^} = 0.5$ . Compute the loss for both possible truths.

Forecast: does the loss depend on whether $y = 0$ or $y = 1$ here?

If $y = 1$ : $ℓ = - lo g (0.5) = 0.693$ nats. Why this step? $y = 1$ keeps the $lo g \overset{y}{^}$ term; substitute $\overset{y}{^} = 0.5$ .
If $y = 0$ : $ℓ = - lo g (1 - 0.5) = - lo g (0.5) = 0.693$ nats. Why this step? $y = 0$ keeps the $lo g (1 - \overset{y}{^})$ term, and $1 - 0.5 = 0.5$ too — so both surviving terms equal $- lo g 0.5$ and the label doesn't matter.
So a coin-flip predictor always pays $ln 2 \approx 0.693$ per example. Why this step? This is the baseline from the parent note — any useful model must beat it.

Verify: $ln 2 = 0.6931$ ; and the blue and pink curves in the figure cross exactly at $\overset{y}{^} = 0.5$ at height $0.693$ . Any model scoring $J > 0.693$ is worse than guessing. ✓

Worked example Ex 5 — Cell: full batch (averaging single losses into

$J$ ) Three examples: $(y = 1, \overset{y}{^} = 0.9)$ , $(y = 0, \overset{y}{^} = 0.2)$ , $(y = 1, \overset{y}{^} = 0.4)$ . Find the batch cost $J$ .

Forecast: which of the three examples contributes the most loss?

$ℓ_{1} = - lo g (0.9) = 0.1054$ (truth 1, term $lo g \overset{y}{^}$ ).
$ℓ_{2} = - lo g (1 - 0.2) = - lo g (0.8) = 0.2231$ (truth 0, term $lo g (1 - \overset{y}{^})$ ).
$ℓ_{3} = - lo g (0.4) = 0.9163$ (truth 1, but only $0.4$ confidence in the true class). Why these steps? Each example uses only the term matching its own label, as in Ex 1–3.
$J = \frac{1}{3} (0.1054 + 0.2231 + 0.9163) = \frac{1.2448}{3} = 0.4149$ nats. Why this step? $J$ is defined here as the average of the per-example $ℓ$ values, so it doesn't grow just because $N$ grows — see the $\frac{1}{N}$ in the parent's boxed formula.

Verify: the hesitant-but-correct example 3 dominates, as expected. And $0.4149 < 0.693$ , so this model already beats the coin-flip baseline of Ex 4. ✓ Feeds straight into Gradient Descent via the $(\overset{y}{^} - y) x$ update.

Worked example Ex 6 — Degenerate cell:

$\overset{y}{^} \to 0$ and $\overset{y}{^} \to 1$ (limits & clipping) What is the loss as the model becomes infinitely sure? Take $y = 1$ .

Forecast: is the loss ever exactly $0$ ? Ever exactly $\infty$ ?

As $\overset{y}{^} \to 1^{-}$ : $ℓ = - lo g (\overset{y}{^}) \to - lo g (1) = 0$ . Why this step? Perfect confidence in the true class → zero loss, but only in the limit; $\overset{y}{^} = 1$ is never actually output by a finite-input sigmoid.
As $\overset{y}{^} \to 0^{+}$ : $ℓ = - lo g (\overset{y}{^}) \to + \infty$ . Why this step? $lo g$ of a number heading to $0$ is a huge negative → loss explodes.
Danger: plugging $\overset{y}{^} = 0$ literally gives $lo g 0 = - \infty \Rightarrow$ NaN. Why this step? Computers can't store $- \infty$ cleanly; the training blows up.
Fix: clip to $[ε, 1 - ε]$ with $ε = 1 0^{- 7}$ . Then the worst single loss for $y = 1$ is $- lo g (1 0^{- 7}) = 16.118$ nats. Why this step? Clipping caps the punishment at a large-but-finite number, keeping arithmetic safe.

Verify: $- lo g (1 0^{- 7}) = 7 ln 10 = 16.118$ . Finite → no NaN. This is the vertical wall you see the blue curve approaching at the left edge of the figure. ✓

Worked example Ex 7 — Real-world word problem

A spam filter is shown 4 emails. Truth (1 = spam): $[1, 0, 1, 0]$ . The filter's spam-probabilities: $[0.99, 0.02, 0.30, 0.85]$ . Score the filter with log-loss and say which email embarrassed it most.

Forecast: email 4 (real mail flagged 85% spam) — will it be the biggest contributor?

Email 1 ( $y = 1$ ): $- lo g (0.99) = 0.01005$ . Confident & right.
Email 2 ( $y = 0$ ): $- lo g (1 - 0.02) = - lo g (0.98) = 0.02020$ . Confident & right.
Email 3 ( $y = 1$ ): $- lo g (0.30) = 1.20397$ . Right side but unsure → medium loss.
Email 4 ( $y = 0$ ): $- lo g (1 - 0.85) = - lo g (0.15) = 1.89712$ . Confident & WRONG → biggest. Why these steps? Spam ( $y = 1$ ) uses $lo g \overset{y}{^}$ ; ham ( $y = 0$ ) uses $lo g (1 - \overset{y}{^})$ .
$J = \frac{1}{4} (0.01005 + 0.02020 + 1.20397 + 1.89712) = \frac{3.13134}{4} = 0.78283$ nats. Why this step? Average the four per-email losses.

Verify: email 4 contributes the most, matching the forecast; and $J = 0.783 > 0.693$ , so despite two great calls, the one confident false alarm drags the filter below the coin-flip baseline. That is log-loss doing its job. ✓

Worked example Ex 8 — Exam twist: units (

$lo g_{2}$ ) + a gradient step (a) Redo the always- $0.5$ baseline in bits (base-2 log). (b) For one example with $z = w^{⊤} x + b$ , $\overset{y}{^} = σ (z) = 0.7$ , true $y = 1$ , and feature $x_{j} = 2$ , compute $\partial ℓ / \partial w_{j}$ .

Forecast: will the base-2 baseline be a rounder number than $0.693$ ?

(a) $ℓ = - lo g_{2} (0.5) = - (- 1) = 1$ bit. Why this step? $lo g_{2}$ just rescales by $1/ ln 2$ ; the natural-log baseline $ln 2$ becomes exactly $1$ bit — the "one coin-flip of information" reading.
(b) From the parent's cancellation, $\partial ℓ / \partial z = \overset{y}{^} - y = 0.7 - 1 = - 0.3$ . Why this step? The messy $\overset{y}{^} (1 - \overset{y}{^})$ from the Sigmoid function cancels, leaving prediction − truth.
Chain rule: $\partial ℓ / \partial w_{j} = (\overset{y}{^} - y) x_{j} = (- 0.3) (2) = - 0.6$ . Why this step? $\partial z / \partial w_{j} = x_{j}$ because $z$ is linear in $w_{j}$ .

Verify: $- lo g_{2} 0.5 = 1$ exactly (round, as forecast). The gradient $- 0.6$ is negative, meaning $w_{j}$ should increase to push $\overset{y}{^}$ toward the true $1$ — exactly the direction Gradient Descent wants. ✓

Forecast-then-Verify recap

Recall Across all cells, which single number is the "pass/fail line"?

$ln 2 \approx 0.693$ nats (or $1$ bit). A batch loss $J$ below it means your model beats random guessing; above it means it's worse than a coin flip.

Recall Why did Ex 2 hurt ~58× more than Ex 1 rewarded, even though the prediction was equally "confident"?

The loss curve $- lo g \overset{y}{^}$ is not symmetric: near $\overset{y}{^} = 1$ it flattens toward $0$ , but near $\overset{y}{^} = 0$ it rockets to $\infty$ . Confident-and-wrong is punished on the steep side. See the blue curve in the figure.

Connections

Sigmoid function — every $\overset{y}{^}$ above came from it, and its derivative is what cancels in Ex 8.
Logistic Regression — the model these losses grade.
Maximum Likelihood Estimation — why the formula is what it is.
Cross-Entropy and KL Divergence — the $lo g_{2}$ "bits" reading in Ex 8.
Gradient Descent — consumes the $(\overset{y}{^} - y) x_{j}$ result of Ex 8.
Mean Squared Error — the loss we rejected in the parent.
Softmax and Categorical Cross-Entropy — the multi-class version of this whole page.