2.6.11 · D5Model Evaluation & Selection
Question bank — Log-loss and calibration
This bank hunts the misconceptions around log-loss and calibration. Every answer gives you the reasoning, not just a verdict. Read the question, cover the answer, commit to a claim, then reveal.
Before the traps, pin down the exact notation so nothing below is ambiguous.
The one picture that explains most traps below is the shape of :
True or false — justify
A model with 99% accuracy must have low log-loss.
False. Accuracy only checks which side of 0.5 each prediction lands; a model can be right 99% of the time yet output 0.51 on every case, giving mediocre log-loss because it earns tiny reward on wins and never expresses real confidence.
Log-loss can be exactly zero.
Only in the limit — it reaches when every prediction is on the true class, but a probability of exactly 1 is never reachable in practice, so real log-loss is a small positive number, never truly .
Log-loss is bounded above by 1.
False. Its range is . A single confident-wrong prediction (, ) sends , so one catastrophic mistake can dominate the whole dataset average.
A perfectly calibrated model is a perfectly accurate model.
False. A model that predicts on a truly 50/50 coin is perfectly calibrated yet only 50% accurate — calibration is about honesty of the number, not correctness of the label.
Softmax outputs are automatically calibrated probabilities.
False. Softmax outputs sum to 1 and sit in , so they look like a distribution, but neural nets are typically overconfident; the logit scale is arbitrary and nothing in standard training forces the numbers to match true frequencies. See Temperature Scaling.
Minimizing log-loss on the training set guarantees good calibration on new data.
False. Log-loss is a proper scoring rule, so it rewards calibration in-distribution, but overparameterized nets can drive training log-loss low while extrapolating badly — measure calibration separately with Reliability Diagrams.
Log-loss and cross-entropy are the same objective.
True. Binary/multi-class log-loss is the negative log-likelihood, which equals the Cross-Entropy Loss between the one-hot true label and the predicted distribution — different names, same formula.
ECE = 0 proves the model is perfectly calibrated.
False. ECE bins predictions, and errors within a bin can cancel (over-confident and under-confident cases averaging out), so a low or zero ECE can hide real miscalibration hiding inside coarse bins.
Two models with identical accuracy always have identical log-loss.
False. Accuracy ignores the magnitude of ; a model that is right confidently and a model that is right timidly score the same accuracy but very different log-loss.
Log-loss punishes an over-confident wrong answer more than an under-confident right answer rewards.
True. The curve is asymmetric and unbounded near 0 (see the figure above), so the penalty for confident-wrong grows without limit while the reward for confident-right is capped near — the whole point is to discourage overconfidence.
Spot the error
"My log-loss dropped, so my probabilities became more trustworthy."
The error: low log-loss ≠ good calibration on unseen data. Log-loss can fall because the model fits the training distribution's frequencies while being wildly overconfident out-of-distribution — verify with ECE or a reliability diagram.
"I'll use accuracy to pick the model for my medical risk score."
The error: accuracy discards probability quality entirely. A risk score needs calibrated probabilities so a stated 20% risk truly means 20%; use log-loss / ECE, not accuracy, for risk-sensitive selection (see Brier Score).
" with just gives a big number, no problem."
The error: is , not a big finite number — that is why implementations clip into before taking the log, to avoid infinite/undefined loss.
"For the loss is ."
The error: for the negative class the model's assigned probability of the observed outcome is , so the loss is . Using rewards the wrong direction.
"I multiplied all my logits by 10 and accuracy stayed the same, so nothing changed."
The error: accuracy is unchanged because the argmax is unchanged, but scaling logits sharpens softmax — probabilities become far more confident, which changes log-loss and calibration dramatically. This is exactly what Temperature Scaling tunes (see the figure below).
"Reliability curve is above the diagonal, so my model is overconfident."
The error: above the diagonal means observed accuracy exceeds stated confidence — that is underconfident (you say 60%, reality is 80%). Overconfident curves sit below the diagonal.
"ECE weights every bin equally, so I average the gaps."
The error: ECE weights each bin by its share of the data, so a 2-sample bin barely counts while a 200-sample bin dominates — a plain average of gaps would let tiny bins distort the result.
Why questions
Why take the negative log instead of just the raw likelihood?
Likelihoods of many samples multiply and shrink toward 0 (underflow); the log turns the product into a stable sum, the negative sign turns "maximize likelihood" into "minimize a loss," and the result is convex per-sample, which suits gradient descent.
Why does log-loss connect to Information Theory specifically?
is the number of bits of surprise (nats if natural log) when a claimed-probability- event occurs; averaging it is the cross-entropy, so log-loss literally measures the extra bits your model spends encoding reality versus a perfect model.
Why bin predictions to measure calibration instead of checking each one?
A single prediction of has one outcome (0 or 1), so you cannot compare "70%" to a frequency of a single event; grouping many predictions lets you compute the observed fraction that were positive and compare it to .
How many bins should I use for ECE, given intra-bin cancellation?
There is a tension: too few bins let over- and under-confident cases inside one bin cancel and hide miscalibration; too many leave each bin with too few samples for a stable accuracy estimate. Practice uses 10–20 equal-width bins, cross-checks with equal-frequency (adaptive) bins so each holds similar counts, and always inspects the full reliability diagram rather than trusting the single ECE number.
Why is calibration "non-negotiable" for medicine and finance but not for a spam filter you only threshold?
If you only need a yes/no decision at a fixed threshold, the ranking (which ROC-AUC captures) suffices; but when the number itself drives a downstream decision — dose, bet size, risk premium — a lie in the probability is a lie in the decision.
Why does log-loss reward calibration at all, mathematically?
Because it is a proper scoring rule: for any true event rate , the expected log-loss is minimized precisely when you report , so the honest, calibrated probability is the optimum — see Proper Scoring Rules.
Why might replacing log-loss with Focal Loss hurt calibration even if it helps rare-class accuracy?
Focal loss down-weights easy, confident examples to focus learning on hard ones, which distorts the probability magnitudes away from true frequencies — great for recall on imbalanced data, but the softmax numbers drift from calibrated values.
How does temperature scaling actually change confidence without changing the prediction?
It divides every logit by a single scalar before softmax. A large shrinks the gaps between logits so softmax flattens toward uniform (less confident); a small exaggerates gaps so it sharpens toward one-hot (more confident). Because dividing all logits by the same never changes their order, the argmax — and thus accuracy — is untouched; only the probabilities move. See the figure below and Temperature Scaling.
Edge cases
What is the log-loss contribution of a prediction regardless of the true label?
It is (i.e. , or 1 bit in base 2) in either case — the model that always shrugs "50/50" pays a fixed, moderate price and never risks the infinite penalty.
What happens to log-loss if but the true label is ?
The loss is — a single perfectly-confident perfectly-wrong prediction makes the entire dataset average infinite, which is why clipping is mandatory.
An empty confidence bin () — what is its ECE contribution?
Zero, because its weight ; empty bins simply drop out of the sum and never affect ECE.
All predictions fall into one bin — is ECE still meaningful?
Barely — with one bin, ECE reduces to the single gap and can look tiny even under severe miscalibration, since intra-bin cancellation is at its worst. Use more bins.
A model outputs identical for every sample and 30% of samples are positive — calibrated?
Yes, perfectly calibrated by definition (stated 30% matches observed 30%) even though it is uselessly non-discriminative — calibration says nothing about the model's ability to separate classes; that is ROC-AUC's job.
What log-loss does a random 50/50 guesser get on a balanced binary set?
About () — this is the natural "no information" baseline; any model scoring worse than is doing worse than coin-flipping.
Recall Quick self-check
Overconfident reliability curve sits where relative to the diagonal? ::: Below it — stated confidence exceeds observed accuracy. Does ECE = 0 guarantee perfect calibration? ::: No — intra-bin cancellation can hide miscalibration. Is a 100%-accurate model necessarily low log-loss? ::: No — timid probabilities (e.g. all 0.51) keep log-loss high despite perfect accuracy. Which log base does this page use? ::: Natural log (base ), so .