Visual walkthrough — Log-loss and calibration

This page rebuilds the parent result one picture at a time. We start with nothing but a coin flip and a guess, and end holding the exact log-loss formula, understanding why every symbol is there. Bring only middle-school arithmetic — everything else we build.

---

Step 1 — A guess and a truth

Let us name the two things carefully, in plain words first.

• The truth is a switch that is either OFF or ON. We write it $y$. We agree $y=0$ means "didn't happen" and $y=1$ means "happened". Nothing more — $y$ is just a label, a flag.
• Your guess is a number between 0 and 1, written $\hat{p}$ (say "p-hat"). The little hat means "this is your estimate, not the truth." $\hat{p}=0.9$ means "I'm 90% sure it happens."

WHY these two and nothing else? Because scoring a forecaster needs exactly two ingredients: what you said ($\hat{p}$) and what reality did ($y$). The whole rest of the page is one question — how do we turn this pair into a fair penalty?

Look at the picture: a number line from 0 to 1 with your guess $\hat{p}$ as a blue dot, and reality shown as a single lit-up bulb (ON = $y=1$). The gap between your dot and the true corner is what we want to punish.

---

Step 2 — The probability you assigned to what actually happened

Two cases, because reality has two doors:

• If it happened ($y=1$): you had put $\hat{p}$ on that door. Your credit is $\hat{p}$.
• If it didn't ($y=0$): the two guesses had to sum to 1, so you had put $1-\hat{p}$ on the "no" door. Your credit is $1-\hat{p}$.

WHY $1-\hat{p}$? Because "happens" and "doesn't happen" cover everything and can't both be true — their probabilities must add to 1. So whatever you gave to YES, the leftover $1-\hat{p}$ automatically went to NO.

The figure splits into two panels — the YES door and the NO door — with the slice of probability you assigned to the door reality actually opened highlighted in yellow. That yellow slice is your credit.

---

Step 3 — One formula for both doors (the clever exponent trick)

Watch what these exponents do:

$\text{credit}=\underset{\text{YES-door term}}{\underbrace{{\hat{p}}^y}}\cdot \underset{\text{NO-door term}}{\underbrace{(1-\hat{p}{)}^{1-y}}}$

${\hat{p}}^y$ :: raise your YES-guess to the power $y$. This term is a light switch. $(1-\hat{p}{)}^{1-y}$ :: raise your NO-guess to the power $1-y$. The other light switch. $\cdot$ :: multiply the two switches together.

WHY does this work? Anything to the power $0$ is $1$ — a "do nothing" factor. So the exponent acts as a selector:

• When $y=1$: first term is ${\hat{p}}^1=\hat{p}$; second is $(1-\hat{p}{)}^0=1$. Product $=\hat{p}$. ✅ matches Step 2.
• When $y=0$: first term is ${\hat{p}}^0=1$; second is $(1-\hat{p}{)}^1=1-\hat{p}$. Product $=1-\hat{p}$. ✅ matches Step 2.

This is the Bernoulli likelihood — the probability of one yes/no outcome under your forecast. (See Cross-Entropy Loss for where this line reappears.)

The picture shows the exponent as a dimmer switch: the power $0$ dims a term all the way to $1$ (invisible), the power $1$ leaves it fully lit. Only the door reality chose stays lit.

---

Step 4 — Turn "good is big" into "penalty is small": take $-\log$

Right now, big credit = good. But a loss should be small = good, so gradient descent can push it down. We need a machine that flips "big is good" into "small is good" — and does three extra favours.

Apply it to the credit from Step 3:

$\ell =-\log ({\hat{p}}^y(1-\hat{p}{)}^{1-y})$

Now use the log rule $\log (a\cdot b)=\log a+\log b$ and $\log (a^c)=c\log a$ — exactly the "products become sums" magic we chose it for:

$\boxed{\ell =-[y\log \hat{p}+(1-y)\log (1-\hat{p})]}$

$y\log \hat{p}$ :: active only when $y=1$ (else $y=0$ kills it). Penalty for the YES door. $(1-y)\log (1-\hat{p})$ :: active only when $y=0$. Penalty for the NO door. $-[\dots ]$ :: the flip that makes "confident-and-right" cost near zero.

The figure plots $-\log$ across $[0,1]$: flat near the right (credit 1 → cost 0) and diving to the sky on the left (credit 0 → cost $\infty$). This cliff is the whole personality of log-loss — it punishes confident wrongness without mercy.

---

Step 5 — Degenerate cases: the edges of the cliff

A formula you cannot trust at its extremes is a trap. Let us walk every corner.

Truth $y$	Guess $\hat{p}$	What you did	$\ell =-\log (\text{credit})$
1	$\to 1$	right & sure	$\to 0$ (no penalty)
1	$\to 0$	wrong & sure	$\to +\infty$ (ruin)
0	$\to 0$	right & sure	$\to 0$ (no penalty)
0	$\to 1$	wrong & sure	$\to +\infty$ (ruin)

WHY the infinities? If you swear $\hat{p}=0$ ("impossible") and it happens, you claimed a real event had no probability. The log of $0$ is $-\infty$; negated, it's $+\infty$. The math is telling you: never say never. In code we clip $\hat{p}$ to $[\epsilon ,1-\epsilon ]$ so a single lunatic prediction doesn't blow the whole average to infinity.

The picture is a square of the four corners; two are painted green (cost $\to 0$), two are painted pink and marked with $\infty$ spikes — the confident-wrong corners where the cliff of Step 4 becomes a wall.

---

Step 6 — From one event to a whole dataset: just add and average

$\mathcal{L}=-\frac{1}{N}{\sum }_{i=1}^N[y_i\log {\hat{p}}_i+(1-y_i)\log (1-{\hat{p}}_i)]$

${\sum }_{i=1}^N$ :: "walk through every sample $i$ and total up its penalty." $\frac{1}{N}$ :: divide by the count so 4 samples and 4 million samples live on the same scale. $y_i,{\hat{p}}_i$ :: the truth and guess for the $i$-th sample.

WHY average instead of just sum? So the number means "typical penalty per prediction" — comparable across datasets of different sizes. This averaged $\mathcal{L}$ is exactly the Cross-Entropy Loss, and it is a genuine proper scoring rule: its minimum is reached only by telling the honest truth.

The bar chart shows each sample's individual penalty as a chalk bar; the tall pink bar (confident-wrong sample) towers over the rest, and the dashed line marks the average $\mathcal{L}$ — visibly dragged up by that one bar.

---

Step 7 — Worked walk-through on 4 points

---

The one-picture summary

The final figure stitches the whole journey: a guess $\hat{p}$ meets a truth $y$ → the exponent selector picks the credit → $-\log$ flips it up the cliff → sum and average gives $\mathcal{L}$. Follow the arrows once and you have re-derived binary log-loss.

Recall Feynman retelling — say it like a friend

You made a bet about a yes/no thing. First we asked: how much of your probability landed on what actually happened? — that's your credit. A slick exponent trick, ${\hat{p}}^y(1-\hat{p}{)}^{1-y}$, writes both the "it happened" and "it didn't" cases in one line, because anything to the power zero just switches itself off. But "big credit is good" is backwards for a loss, so we hit it with negative-log — a machine that gives zero pain when you were sure-and-right and infinite pain when you were sure-and-wrong, and conveniently turns the multiplying of probabilities into simple adding. We checked all four corners so nothing surprises us, agreed to never literally say 0 or 1 (or we'd get infinity), then added everyone's pain and divided by how many there were. That average is log-loss. It rewards honesty above all — the way to make it smallest is to say numbers that are actually true.

Recall Quick checks

Why negative log and not, say, $1-\text{credit}$? ::: Only $\log$ turns products into sums, so many predictions' penalties add cleanly — and it makes the confident-wrong penalty explode, which is the behaviour we want. What does the exponent $y$ in ${\hat{p}}^y$ actually do? ::: It's a selector: power 0 makes a term equal 1 (off), power 1 leaves it on. So only the door reality opened contributes. Why can log-loss be infinite? ::: If you predict exactly 0 for something that happens, you claimed an impossible event occurred; $-\log 0=+\infty$. Fix by clipping to $[\epsilon ,1-\epsilon ]$. Where is calibration in all this? ::: Log-loss scores sharpness+honesty per point; calibration (Reliability Diagrams, Brier Score) separately checks that your stated 70% really happens 70% of the time.

Related: Information Theory (log-loss = expected bits of surprise), Temperature Scaling and Focal Loss (post-hoc and training-time fixes), ROC-AUC (ranking, ignores calibration), Bayesian Neural Networks (calibrated uncertainty by construction).