Log-loss and calibration: Worked examples — AI-ML deep dive

This page is the exhaustive drill. The parent note 2.6.11 gave you the formulas. Here we throw every kind of input at those formulas — perfect predictions, catastrophic ones, the edge cases that produce $0$ or $\infty$ , the degenerate model, and an exam trap — and grind each one to a number.

Before any formula appears we re-anchor its meaning in one line, because the promise of this vault is: no symbol used before it is earned.

Recall Two symbols we lean on constantly

$\overset{p}{^}$ ::: the probability your model announced for the event happening (a number between 0 and 1). $y$ ::: the truth, written as $1$ if the event happened and $0$ if it did not. Log-loss for one sample ::: $- lo g \overset{p}{^}$ when $y = 1$ , and $- lo g (1 - \overset{p}{^})$ when $y = 0$ — "surprise at the truth".

The scenario matrix

Every log-loss / calibration problem is one of these case classes. The examples below each carry a tag like [Cell A] so you can see the whole space is covered.

Cell	Case class	What makes it special	Example
A	Confident & correct	$\overset{p}{^}$ near the true label → loss near $0$	Ex 1
B	Confident & wrong	$\overset{p}{^}$ near the opposite of truth → loss explodes	Ex 2
C	Timid / uncertain ( $\overset{p}{^} \approx 0.5$ )	moderate loss regardless of outcome	Ex 3
D	Degenerate input ( $\overset{p}{^} = 0$ or $1$ )	$lo g 0 = - \infty$ : the clipping fix	Ex 4
E	Multi-class (one-hot)	only the true class term survives	Ex 5
F	Calibration: over- vs under-confident bins	ECE via reliability gap	Ex 6
G	Degenerate calibration (empty bin, perfect bin)	how ECE handles $0/0$ and gap $= 0$	Ex 7
H	Real-world word problem	translate a story → $\overset{p}{^}, y$	Ex 8
I	Exam twist (accuracy vs log-loss disagree)	which model "wins"?	Ex 9

Two anchor pictures for the whole page:

The loss curves above are the single most important image on this page. Read the amber curve as "the penalty when the truth was $y = 1$ ": it dives to $0$ as your announced $\overset{p}{^} \to 1$ (you were right and confident), and rockets toward $+ \infty$ as $\overset{p}{^} \to 0$ (you were confidently wrong). The cyan curve is the mirror for $y = 0$ .

The reliability diagram above is the picture behind every calibration example: the diagonal is "confidence tells the truth"; a bar's height above/below the diagonal is its calibration gap.

The worked examples

Worked example Example 4 — Degenerate input:

$\overset{p}{^} = 0$ or $1$ [Cell D] A model outputs $\overset{p}{^} = 0$ for a case that turns out positive ( $y = 1$ ). What is the loss?

Forecast: finite or infinite?

Branch: $y = 1 \Rightarrow - lo g \overset{p}{^} = - lo g 0$ . Why this step? $lo g 0 = - \infty$ , so the loss is $+ \infty$ — a categorical "impossible" that happened.
The engineering fix: clip predictions to $[ε, 1 - ε]$ with $ε = 1 0^{- 15}$ . Why this step? Real code (e.g. sklearn.log_loss) never divides by zero; it replaces $0$ with a tiny $ε$ so the number is huge-but-finite instead of NaN.
Clipped value: $- lo g (1 0^{- 15}) = 34.54$ . Why this step? $- lo g (1 0^{- 15}) = 15 ln 10 = 15 \times 2.3026 = 34.54$ .

Verify: $34.54$ is enormous but finite — a single such sample can dominate an entire batch's average, which is exactly the warning. Never let a model emit a hard $0$ or $1$ . ✅

Worked example Example 6 — Calibration with over- and under-confident bins

[Cell F] 40 predictions in two bins:

Bin	Count	conf	acc
high	30	0.90	0.70
low	10	0.20	0.40

Forecast: which bin hurts ECE more — the big overconfident one or the small underconfident one?

Per-bin gaps: high $= ∣0.70 - 0.90∣ = 0.20$ ; low $= ∣0.40 - 0.20∣ = 0.20$ . Why this step? ECE cares about the absolute gap, not direction.
Weight by fraction of data: $ECE = \frac{30}{40} (0.20) + \frac{10}{40} (0.20)$ . Why this step? A bin with 30 samples should count 3× a bin with 10 — big bins carry more evidence.
Compute: $0.75 (0.20) + 0.25 (0.20) = 0.15 + 0.05 = 0.20$ .

Verify: ECE $= 0.20$ , i.e. 20% average miscalibration. The high bin (below diagonal in s02) is overconfident, the low bin (above diagonal) is underconfident. A temperature fix would flatten the overconfident one. ✅

Worked example Example 7 — Degenerate calibration: empty bin & perfect bin

[Cell G] Ten bins, but only two contain data:

Bin	Count	conf	acc
0.4–0.6	50	0.50	0.50
0.8–1.0	50	0.90	0.80
(other 8 bins)	0	—	—

Forecast: do the 8 empty bins add to ECE? Does the perfect bin?

Empty bins: weight is $\frac{∣ B _{m} ∣}{N} = \frac{0}{100} = 0$ . Why this step? A bin with no samples has zero weight, so its undefined acc (a $0/0$ ) is multiplied away — it can never contribute. This is why we never compute acc for empty bins.
Perfect bin (0.4–0.6): gap $= ∣0.50 - 0.50∣ = 0$ , contributes $0$ . Why this step? Perfect calibration in a bin means zero gap regardless of size.
Only the last bin matters: $ECE = \frac{50}{100} ∣0.80 - 0.90∣ = 0.5 (0.10) = 0.05$ .

Verify: ECE $= 0.05$ (5%). Confirms two degenerate rules: empty bins are silent, perfectly-calibrated bins are silent. Only real gaps count. ✅

Worked example Example 8 — Real-world word problem

[Cell H] A weather app says " $80%$ chance of rain" on five days. It actually rained on 3 of them. (a) What's the log-loss? (b) Is the app calibrated?

Forecast: guess whether the app is over- or under-confident before computing.

Translate the story: each day $\overset{p}{^} = 0.8$ ; three days have $y = 1$ (rain), two have $y = 0$ . Why this step? " $80%$ chance" is $\overset{p}{^}$ ; "it rained" is $y$ .
Rainy days ( $y = 1$ ): loss $- lo g 0.8 = 0.2231$ each, ×3 $= 0.6694$ .
Dry days ( $y = 0$ ): loss $- lo g (1 - 0.8) = - lo g 0.2 = 1.6094$ each, ×2 $= 3.2189$ .
Average over 5: $L = (0.6694 + 3.2189) /5 = 0.7777$ . Why this step? Divide the total surprise by the number of days.
Calibration: it rained $3/5 = 0.60$ of the time, but confidence was $0.80$ . Why this step? Observed frequency vs stated confidence is the reliability check.

Verify: Gap $= ∣0.60 - 0.80∣ = 0.20$ , so with one bin ECE $= 0.20$ . The app is overconfident (said 80%, delivered 60%) — matches the forecast for an app that over-promises rain. Loss $0.7777$ is above the $0.693$ coin-flip line, confirming the miscalibration hurts. ✅

Worked example Example 9 — Exam twist: accuracy vs log-loss disagree

[Cell I] Two models on the same 2 samples (both truly positive, $y = 1$ ), threshold $0.5$ :

Model A: $\overset{p}{^} = [0.51, 0.51]$ → both classified positive → accuracy 100%.
Model B: $\overset{p}{^} = [0.90, 0.30]$ → one above, one below $0.5$ → accuracy 50%.

Forecast: by accuracy A wins. Which wins by log-loss?

Model A loss: $- lo g (0.51)$ each $= 0.6733$ ; average $= 0.6733$ . Why this step? A is barely committing — every prediction sits near the coin-flip line.
Model B loss: $- lo g (0.90) = 0.1054$ and $- lo g (0.30) = 1.2040$ ; average $= (0.1054 + 1.2040) /2 = 0.6547$ . Why this step? B's confident correct call (0.90) is cheap; its confident-ish wrong call (0.30) is dear — but they average lower than A.

Verify: $L_{B} = 0.6547 < L_{A} = 0.6733$ . Log-loss prefers B even though B has lower accuracy, because B produces informative probabilities while A hugs $0.5$ . This is the parent note's "Example 3" made numeric — accuracy and probability quality can disagree. Compare with Brier Score and Proper Scoring Rules, which reward the same honesty. ✅

Recall

Recall Which branch of the log-loss do I use?

When $y = 1$ ::: use $- lo g \overset{p}{^}$ . When $y = 0$ ::: use $- lo g (1 - \overset{p}{^})$ .

Recall Why does one confidently-wrong sample dominate a batch?

Because $- lo g$ of a tiny number is huge (unbounded above) ::: while a correct confident sample only earns a penalty near $0$ — the curve is steep on the wrong side, flat on the right side.

Recall Do empty calibration bins change ECE?

No ::: their weight $∣ B_{m} ∣/ N = 0$ multiplies away any (undefined) gap.

Recall Can a lower-accuracy model have lower log-loss?

Yes (Ex 9) ::: log-loss rewards honest, informative probabilities; accuracy only checks the side of the $0.5$ threshold.

Related deep tools: Reliability Diagrams, Temperature Scaling, Focal Loss, ROC-AUC, Bayesian Neural Networks.