Visual walkthrough — Log-loss - binary cross-entropy

We will use only three colours in the figures: white for the raw geometry, cyan for the quantity we are currently building, and amber for the punchline of each step. Watch the amber.

---

Step 1 — A single guess is a number between 0 and 1

WHAT. A logistic model looks at an example and outputs one number, $\hat{y}$ (say "y-hat"). It is a probability: the model's stated chance that the true label is $1$.

WHY start here. Before we can score a guess, we must be crystal-clear on what a guess is. It is not a category ("cat"/"dog"); it is a slider between $0$ and $1$. The label $y$ itself is only ever $0$ or $1$ — a hard fact. The gap between the soft $\hat{y}$ and the hard $y$ is the entire story.

PICTURE. The slider below runs from $0$ (certain the answer is $0$) to $1$ (certain the answer is $1$). The cyan dot is where the model placed itself.

---

Step 2 — One formula that reports the probability of the actual outcome

WHAT. We want a single expression that, given the model's slider $\hat{y}$, tells us the probability the model assigned to whatever actually happened.

WHY. If the truth is $y=1$, the relevant number is $\hat{y}$. If the truth is $y=0$, the relevant number is $1-\hat{y}$ (the leftover chance the slider kept for outcome $0$). We do not want two formulas glued with an "if". One clean formula is:

$P(y\mid \hat{y})=\underset{\text{prob of outcome }1}{\underbrace{\hat{y}}}{}^{\stackrel{\text{on iff }y=1}{\overbrace{y}}}\cdot \underset{\text{prob of outcome }0}{\underbrace{(1-\hat{y})}}{}^{\stackrel{\text{on iff }y=0}{\overbrace{1-y}}}$

The trick is the exponent. Anything to the power $0$ is $1$ (it disappears), anything to the power $1$ is itself (it stays). So the exponent is a switch: exactly one factor survives.

PICTURE. Two arrows. When $y=1$ the switch turns on the $\hat{y}$ arrow; when $y=0$ it turns on the $1-\hat{y}$ arrow. This is the [[Sigmoid function|]]-fed Bernoulli model.

---

Step 3 — Many examples multiply (that's what independence means)

WHAT. Real data is $N$ examples, not one. Give each its own slider ${\hat{y}}_i$ and label $y_i$. The chance the model produces the whole dataset exactly as observed is the product of the individual chances.

WHY a product. "Independent" means one example's outcome does not lean on another's. The probability of two independent things both happening is the product (like getting heads AND heads: $\frac{1}{2}\cdot \frac{1}{2}$). So:

$L={\prod }_{i=1}^N\underset{\text{Step 2 for example }i}{\underbrace{{\hat{y}}_i^{y_i}(1-{\hat{y}}_i{)}^{1-y_i}}}$

Here $\prod$ (capital pi) just means "multiply all of these together", the way $\sum$ means "add them all". This $L$ is the likelihood: how believable the whole dataset is under the current model. The best model is the one making $L$ biggest — this is Maximum Likelihood.

PICTURE. A chain of per-example bricks multiplied into one tall stack. Notice how the stack's height collapses toward $0$ the moment any single brick is tiny — one confident mistake tanks the whole product.

---

Step 4 — Take the log: products become sums, tiny numbers become manageable

WHAT. Apply $\log$ (natural log, base $e$) to $L$. Because $\log (ab)=\log a+\log b$, every product turns into a sum:

$\log L={\sum }_{i=1}^N[\underset{\text{survives if }{y}_i=1}{\underbrace{y_i\log {\hat{y}}_i}}+\underset{\text{survives if }{y}_i=0}{\underbrace{(1-y_i)\log (1-{\hat{y}}_i)}}]$

The exponents from Step 2 came down in front of the logs (that's the rule $\log (a^k)=k\log a$), turning switches-in-exponents into switches-as-multipliers.

WHY log, of all tools? Three reasons, each answering "why not something else":

1. Turns product into sum — sums are easy to differentiate term by term; products aren't.
2. Monotone — $\log$ never re-orders; whatever maximizes $L$ also maximizes $\log L$. We lose no information about where the best model is.
3. Rescues tiny numbers — a product of thousands of probabilities underflows to $0$ on a computer; its log stays a comfortable finite number.

PICTURE. The tall multiplicative stack from Step 3 is flattened into a lineup of additive bars — same information, friendlier shape. Each $\log {\hat{y}}_i$ is negative (log of a number below $1$), shown dipping below the axis.

---

Step 5 — Flip the sign and average: the loss is born

WHAT. Optimizers like to minimize. Maximizing $\log L$ = minimizing $-\log L$. Divide by $N$ so the number doesn't grow just because you added more data:

$J=-\frac{1}{N}{\sum }_{i=1}^N[y_i\log {\hat{y}}_i+(1-y_i)\log (1-{\hat{y}}_i)]$

Term by term: the $-\frac{1}{N}$ is "make it a minimizable average"; the bracket is the per-example log from Step 4; the sum runs over the dataset.

WHY negate. $\log (\text{something}<1)$ is negative, so $\log L$ is a pile of negatives. The leading minus flips the whole pile positive — a loss you drive down toward $0$. This is exactly the cross-entropy between the true label and the prediction.

PICTURE. The one-example curve $\ell (\hat{y})$ for a fixed truth. When $y=1$, loss is near $0$ at $\hat{y}=1$ and shoots to $+\infty$ as $\hat{y}\to 0$. When $y=0$, it's the mirror image. Amber marks the "confident-and-wrong" cliff.

---

Step 6 — Every corner case, shown explicitly

WHAT. We check all four regions of the (truth, guess) grid plus the degenerate endpoints, so no reader meets an unshown scenario.

WHY. A loss you don't trust at the edges is a loss you don't trust. Let's walk them.

• Truth $1$, guess high ($\hat{y}\to 1$): $-\log \hat{y}\to 0$. Humble-and-right → cheap.
• Truth $1$, guess low ($\hat{y}\to 0$): $-\log \hat{y}\to +\infty$. Confident-and-wrong → catastrophic.
• Truth $0$, guess low ($\hat{y}\to 0$): $-\log (1-\hat{y})\to 0$. Cheap again (mirror image).
• Truth $0$, guess high ($\hat{y}\to 1$): $-\log (1-\hat{y})\to +\infty$. The other cliff.
• The fence-sitter ($\hat{y}=0.5$): cost is $-\log 0.5=\ln 2\approx 0.693$ for every label. This is the baseline any real model must beat.
• Degenerate endpoints ($\hat{y}=0$ or $1$ exactly): $\log 0=-\infty$ → NaN. Forbidden. In code, clip to $[\epsilon ,1-\epsilon ]$.

PICTURE. The full $2\times 2$ grid of quadrants, amber on the two "confident-and-wrong" corners, cyan on the two "right" corners, and the grey $\ln 2$ waterline running through the middle.

---

Step 7 — Differentiate, and watch the mess cancel

WHAT. Set $z=w^{\top }x+b$ and $\hat{y}=\sigma (z)=\frac{1}{1+e^{-z}}$. The sigmoid has the gift ${\sigma }^{\prime }(z)=\hat{y}(1-\hat{y})$. Differentiate $\ell$ with respect to $z$ using the chain rule:

$\frac{\partial \ell }{\partial \hat{y}}=\underset{\text{from }y\log \hat{y}}{\underbrace{-\frac{y}{\hat{y}}}}+\underset{\text{from }(1-y)\log (1-\hat{y})}{\underbrace{\frac{1-y}{1-\hat{y}}}}=\frac{\hat{y}-y}{\hat{y}(1-\hat{y})}$

Then multiply by $\frac{d\hat{y}}{dz}=\hat{y}(1-\hat{y})$:

$\frac{\partial \ell }{\partial z}=\frac{\hat{y}-y}{\hat{y}(1-\hat{y})}\cdot \hat{y}(1-\hat{y})=\boxed{\hat{y}-y}$

WHY this is the payoff. The ugly $\hat{y}(1-\hat{y})$ in the denominator is exactly the ${\sigma }^{\prime }$ we multiply by — they annihilate. The gradient is just prediction − truth, precisely like linear regression, but without MSE's vanishing-gradient disease. When the model is confidently wrong, $\hat{y}-y$ stays large, so gradient descent pushes hard right when you need it most.

PICTURE. Two curves overlaid: the amber ${\sigma }^{\prime }(z)$ (which flatlines to $0$ when confidently wrong — MSE's trap) and the cyan $\hat{y}-y$ (which stays a healthy size). The cancellation is drawn as the amber factor being crossed out.

Recall Why doesn't MSE get this cancellation?

MSE's derivative carries a lone ${\sigma }^{\prime }(z)=\hat{y}(1-\hat{y})$ with nothing to cancel it. When $z$ is large and wrong, ${\sigma }^{\prime }\to 0$, so the gradient dies — learning stalls. Log-loss's $-\log$ produces a $1/[\hat{y}(1-\hat{y})]$ that eats the ${\sigma }^{\prime }$. That's the whole reason we switched.

---

The one-picture summary

One guess → Bernoulli probability → multiply across data → log to a sum → negate and average → the log-loss surface → differentiate to $\hat{y}-y$. Every arrow above is a step you just walked.

Recall Feynman: the whole walkthrough in plain words

The model puts a bet on a slider from 0 to 1 for "is this a 1?". I have one honest formula that reads off how much probability the model actually gave to what really happened. For a whole dataset of independent examples I multiply all those probabilities — but products of tiny numbers vanish, so I take the log, which turns the multiply into an add and behaves nicely. Bigger is better for "how believable is the data", so to make something to minimize I flip the sign and divide by how many examples there are. The resulting curve is gentle when I'm humble-and-right and a vertical cliff when I'm confident-and-wrong — never bet the farm on a mistake. Finally, when I ask "which way should I nudge the weights?", all the messy sigmoid factors cancel and I'm left with the cleanest possible answer: guess minus truth. That's log-loss, birth to gradient.

---

Connections

• Log-loss - binary cross-entropy — the parent note this page derives in pictures.
• Sigmoid function — supplies $\hat{y}=\sigma (z)$ and the ${\sigma }^{\prime }=\hat{y}(1-\hat{y})$ identity that cancels in Step 7.
• Maximum Likelihood Estimation — the "make the data most likely" principle behind Steps 3–5.
• Cross-Entropy and KL Divergence — what $J$ is, seen from information theory.
• Gradient Descent — consumes the clean $\hat{y}-y$ gradient.
• Mean Squared Error — the loss we rejected, and why its gradient stalls.
• Logistic Regression — the model this loss trains.
• Softmax and Categorical Cross-Entropy — the multi-class sequel to this derivation.