Log-loss and calibration: Exercises — AI-ML deep dive

Every logarithm on this page is the natural log ( $ln$ , base $e \approx 2.718$ ) unless we say otherwise. That is the ML default. When base changes the answer, we call it out.

Level 1 — Recognition

Exercise 1.1

For a single binary sample with true label $y = 1$ and predicted probability $\overset{p}{^} = 0.9$ , which expression gives its log-loss?

(a) $- lo g (1 - \overset{p}{^})$ (b) $- lo g (\overset{p}{^})$ (c) $\overset{p}{^} - y$ (d) $(\overset{p}{^} - y)^{2}$

Recall Solution

The per-sample binary log-loss is $ℓ = - [y lo g \overset{p}{^} + (1 - y) lo g (1 - \overset{p}{^})] .$ With $y = 1$ , the second term vanishes ( $1 - y = 0$ ), leaving $- lo g \overset{p}{^}$ . Answer: (b). Numerically $- lo g (0.9) = 0.10536$ . Option (d) is the Brier per-sample term (see Brier Score), not log-loss.

Exercise 1.2

A reliability diagram plots a model's curve below the diagonal $y = x$ everywhere. Is the model over- or under-confident?

Recall Solution

The x-axis is confidence (what you claimed), the y-axis is accuracy (what happened). Below the diagonal means accuracy $<$ confidence: you said 80%, reality delivered 60%. That is overconfidence. Look at Figure s01 — the orange curve sags under the navy diagonal.

Exercise 1.3

State the range of ECE and what value means "perfectly calibrated."

Recall Solution

$ECE \in [0, 1]$ . It is a weighted average of absolute gaps $∣ acc - conf ∣$ , each in $[0, 1]$ , so the average stays in $[0, 1]$ . ECE $= 0$ means every bin's accuracy equals its confidence — perfect calibration.

Level 2 — Application

Exercise 2.1

Compute the log-loss of these three predictions (natural log).

Sample	$y$	$\overset{p}{^}$
1	1	0.8
2	0	0.3
3	1	0.4

Recall Solution

Per sample, use $- lo g \overset{p}{^}$ when $y = 1$ and $- lo g (1 - \overset{p}{^})$ when $y = 0$ : $ℓ_{1} = - lo g (0.8) = 0.22314, ℓ_{2} = - lo g (0.7) = 0.35667, ℓ_{3} = - lo g (0.4) = 0.91629.$ Average: $L = \frac{0.22314 + 0.35667 + 0.91629}{3} = 0.49870.$ Sample 3 (right class, but only 40% confident) dominates — the model was timid on a correct answer.

Exercise 2.2

A 3-class problem. True class is index 2 (one-hot $y = [0, 1, 0]$ ). Predicted $\hat{p} = [0.2, 0.5, 0.3]$ . Give the log-loss.

Recall Solution

Multi-class cross-entropy collapses to the log of the true class's probability only, because the one-hot zeros out the others: $L = - \sum_{k} y_{k} lo g \overset{p}{^}_{k} = - lo g (0.5) = 0.69315.$ Notice this equals $lo g 2$ — a "coin-flip" amount of surprise. See Information Theory for why $lo g 2$ is exactly one bit.

Exercise 2.3

Five predictions fall in one bin. Confidences ${0.7, 0.7, 0.8, 0.6, 0.7}$ ; outcomes $y = {1, 0, 1, 1, 1}$ . Find $conf$ , $acc$ , and the bin's gap.

Recall Solution

$conf = \frac{0.7 + 0.7 + 0.8 + 0.6 + 0.7}{5} = \frac{3.5}{5} = 0.70,$ $acc = \frac{1 + 0 + 1 + 1 + 1}{5} = \frac{4}{5} = 0.80.$ Gap $= ∣ acc - conf ∣ = ∣0.80 - 0.70∣ = 0.10$ . Here accuracy beats confidence, so this bin is underconfident.

Level 3 — Analysis

Exercise 3.1

Model X predicts $\overset{p}{^} = 0.99$ and the truth is $y = 0$ . Model Y predicts $\overset{p}{^} = 0.6$ , truth $y = 0$ . Both are wrong by the 0.5 threshold. Compare their log-losses and explain the asymmetry.

Recall Solution

Both have $y = 0$ , so loss $= - lo g (1 - \overset{p}{^})$ : $ℓ_{X} = - lo g (0.01) = 4.60517, ℓ_{Y} = - lo g (0.4) = 0.91629.$ Ratio $\approx 5 \times$ . Same "wrong" verdict, wildly different penalty. This is the whole point of log-loss: it grades your confidence, not just your label. The $- lo g$ curve is nearly flat near the correct end and shoots to $\infty$ as your assigned probability for the true event heads to 0 — see Figure s02.

Exercise 3.2

Example 2 in the parent note reported $ECE = 0.055$ . Recompute it, then state which bin contributes the most and why.

Bin	Count	conf	acc
0.0–0.2	10	0.15	0.10
0.2–0.4	20	0.30	0.25
0.4–0.6	30	0.50	0.50
0.6–0.8	25	0.70	0.60
0.8–1.0	15	0.90	0.80

Recall Solution

$ECE = \sum_{m} \frac{∣ B _{m} ∣}{N} ∣ acc - conf ∣$ with $N = 100$ : $0.10 (0.05) + 0.20 (0.05) + 0.30 (0) + 0.25 (0.10) + 0.15 (0.10)$ $= 0.005 + 0.010 + 0 + 0.025 + 0.015 = 0.055.$ Biggest contributor: the 0.6–0.8 bin (0.025). It combines a large weight ( $25/100$ ) with a full $0.10$ gap. The 0.4–0.6 bin, though largest, contributes nothing — it is perfectly calibrated. Lesson: ECE rewards big and accurate bins, punishes big and miscalibrated ones.

Exercise 3.3

A model outputs logits $[1, 2, 3]$ . Someone scales all logits by $T = 0.5$ (temperature scaling with $T = 2$ means dividing logits by 2, i.e. multiplying by $0.5$ ). Does the argmax prediction change? What happens to confidence?

Recall Solution

Softmax of $[1, 2, 3]$ : exponentials $e^{1}, e^{2}, e^{3} = 2.71828, 7.38906, 20.0855$ , sum $30.1928$ , so $\hat{p} = [0.09003, 0.24473, 0.66524] .$ After scaling to $[0.5, 1, 1.5]$ : $e^{0.5}, e^{1}, e^{1.5} = 1.64872, 2.71828, 4.48169$ , sum $8.84869$ , giving $\hat{p}_{T} = [0.18632, 0.30720, 0.50648] .$ The top class stays index 3 (argmax unchanged — softmax is monotone in the logits), but its confidence drops from $0.665$ to $0.506$ . Temperature softens the distribution without touching accuracy — exactly why it is a pure post-hoc calibration knob.

Level 4 — Synthesis

Exercise 4.1

For a single sample with $y = 1$ , compare log-loss $- lo g \overset{p}{^}$ against Brier $(\overset{p}{^} - 1)^{2}$ at $\overset{p}{^} \in {0.9, 0.5, 0.1}$ . Which grows faster as $\overset{p}{^} \to 0$ , and what does that imply for training?

Recall Solution

$\overset{p}{^}$	log-loss $- lo g \overset{p}{^}$	Brier $(\overset{p}{^} - 1)^{2}$
0.9	0.10536	0.01
0.5	0.69315	0.25
0.1	2.30259	0.81
As $\overset{p}{^} \to 0$ the Brier term saturates at $1$ , but log-loss $\to \infty$ . So log-loss produces much larger gradients on confidently-wrong samples, driving the optimizer harder to fix them. Both are proper scoring rules; log-loss just has an unbounded, steeper tail — one reason it is the default training loss while Brier is often used as a diagnostic.

Exercise 4.2

Design a 2-sample dataset where Model A has lower accuracy but lower (better) log-loss than Model B. Give explicit numbers.

Recall Solution

Let both samples have $y = 1$ .

Model A: predicts $\overset{p}{^} = 0.6$ on both. Both correct by threshold $0.5$ ? Yes, both $> 0.5$ . Accuracy $= 100%$ . Hmm — we want A lower accuracy, so tweak:
Model A: $\overset{p}{^} = {0.4, 0.9}$ . Threshold-0.5 predictions: sample 1 predicted 0 (wrong), sample 2 predicted 1 (right). Accuracy = 50%. Log-loss $= \frac{1}{2} [- lo g 0.4 - lo g 0.9] = \frac{1}{2} [0.91629 + 0.10536] = 0.51083$ .
Model B: $\overset{p}{^} = {0.51, 0.51}$ . Both predicted 1 (right). Accuracy = 100%. Log-loss $= \frac{1}{2} [- lo g 0.51 - lo g 0.51] = - lo g 0.51 = 0.67334$ .

Model A: lower accuracy (50% vs 100%) yet lower log-loss (0.511 vs 0.673), because its one confident-correct call (0.9) outweighs its near-miss. This mirrors Example 3 in the parent: accuracy and probability quality are different axes.

Exercise 4.3

Two independent predictions have per-sample losses $ℓ_{1}$ and $ℓ_{2}$ . Show that the total log-loss equals $- lo g$ of the joint likelihood, and explain why this is why we chose the logarithm.

Recall Solution

Independent events multiply: joint likelihood $L = L_{1} \cdot L_{2}$ where $L_{i}$ is the probability the model gave to the actual outcome $i$ . Then $- lo g L = - lo g (L_{1} L_{2}) = - lo g L_{1} - lo g L_{2} = ℓ_{1} + ℓ_{2} .$ The log turns a product of likelihoods into a sum of losses. That additivity is exactly what lets us average over a dataset and take clean per-sample gradients. It also makes each $ℓ_{i}$ read as bits/nats of surprise — the Information Theory interpretation. This is the mathematical reason the log (and not, say, the raw likelihood) is used.

Level 5 — Mastery

Exercise 5.1

Prove that for a single fixed true probability $q = P (y = 1)$ , the expected log-loss $- [q lo g \overset{p}{^} + (1 - q) lo g (1 - \overset{p}{^})]$ is minimized at $\overset{p}{^} = q$ . (This is the "properness" of log-loss.)

Recall Solution

Define $f (\overset{p}{^}) = - [q lo g \overset{p}{^} + (1 - q) lo g (1 - \overset{p}{^})]$ . Differentiate: $f^{'} (\overset{p}{^}) = - \frac{q}{p ^} + \frac{1 - q}{1 - p ^} .$ Set to zero: $\frac{q}{p ^} = \frac{1 - q}{1 - p ^} \Rightarrow q (1 - \overset{p}{^}) = (1 - q) \overset{p}{^} \Rightarrow q - q \overset{p}{^} = \overset{p}{^} - q \overset{p}{^} \Rightarrow q = \overset{p}{^} .$ Second derivative $f^{''} (\overset{p}{^}) = \frac{q}{p ^ ^{2}} + \frac{1 - q}{( 1 - p ^ ) ^{2}} > 0$ , so this critical point is a minimum. Therefore truthful reporting $\overset{p}{^} = q$ is optimal — log-loss is a strictly proper scoring rule. Figure s03 shows the bowl for $q = 0.7$ bottoming exactly at $0.7$ .

Exercise 5.2

The minimum value of the expected loss at $\overset{p}{^} = q$ is the entropy $H (q) = - q lo g q - (1 - q) lo g (1 - q)$ . Evaluate it at $q = 0.7$ and interpret.

Recall Solution

$H (0.7) = - 0.7 lo g 0.7 - 0.3 lo g 0.3 = - 0.7 (- 0.35667) - 0.3 (- 1.20397)$ $= 0.24967 + 0.36119 = 0.61086 nats .$ Even a perfectly honest model cannot drive log-loss to zero when outcomes are genuinely random ( $q = 0.7$ ): the floor is the entropy $0.611$ nats — the irreducible surprise of the process. Any log-loss above this is the model's fault; the gap is the KL divergence from truth to prediction.

Exercise 5.3

A calibrated model on a balanced binary task predicts a constant $\overset{p}{^} = 0.5$ for every sample. What is its log-loss, and why can a smarter model beat it only if the task carries real signal?

Recall Solution

With $\overset{p}{^} = 0.5$ every term is $- lo g 0.5 = 0.69315$ (this is $lo g 2$ , exactly 1 bit). So $L = 0.69315$ regardless of labels. A model beats this only by assigning $\overset{p}{^}$ away from $0.5$ in the right direction — pushing toward 1 on true positives, toward 0 on true negatives. That requires the features to actually predict $y$ (real signal). On pure noise ( $q = 0.5$ everywhere), Exercise 5.1 says the optimal $\overset{p}{^}$ is $0.5$ , so $0.69315$ nats is the unbeatable floor. This constant model is the classic log-loss baseline; any real model must undercut it to justify itself.

Recall Quick self-check

Per-sample loss for $y = 1$ ::: $- lo g \overset{p}{^}$ Per-sample loss for $y = 0$ ::: $- lo g (1 - \overset{p}{^})$ ECE weights each bin by ::: its fraction of samples $∣ B_{m} ∣/ N$ Temperature scaling changes accuracy? ::: No — argmax is preserved Minimum expected log-loss for random $q$ ::: the entropy $H (q)$ , not 0

Related: Parent: Log-loss & Calibration · Brier Score · ROC-AUC · Focal Loss · Temperature Scaling · Reliability Diagrams · Proper Scoring Rules · Information Theory · Cross-Entropy Loss · Bayesian Neural Networks