Logistic regression — sigmoid, cross-entropy loss: Visual walkthrough

This page rebuilds the entire engine of logistic regression as a sequence of pictures. We start from a single yes/no question and end at the one gradient formula that trains everything. Every symbol is drawn before it is used.

Step 1 — A question with only two answers

WHAT. Imagine one turbine blade. We ask a single question: will it fail? There are only two answers: yes (we write this as the number $y = 1$ ) and no (we write $y = 0$ ).

WHY. Everything downstream needs a target to aim at. Using the numbers $1$ and $0$ (instead of the words "yes"/"no") lets us do arithmetic on the answer later — subtract it, multiply it, average it.

PICTURE. Two dots on a vertical axis: a red dot at height $1$ ("fail"), a black dot at height $0$ ("safe"). There is nothing in between — a real blade either failed or it didn't.

Step 2 — The raw score $z$ lives on the whole number line

WHAT. Our model looks at a feature $x$ (say, operating hours) and computes one plain number: $z = how much x matters w \cdot the measurement x + baseline shift b$

WHY. This is the only thing a machine finds cheap: multiply and add. But notice the disaster — $z$ can be anything: $- 40$ , $0$ , $+ 7000$ . It is not a probability. A probability must sit between $0$ and $1$ . We have a mismatch to fix.

PICTURE. A horizontal number line stretching from $- \infty$ to $+ \infty$ with $z$ sliding freely along it. The red segment marks the only region a probability is allowed to live in — a tiny sliver from $0$ to $1$ — showing how badly $z$ overshoots.

Step 3 — Why odds is the natural bridge

WHAT. We connect the free number $z$ to a probability $p$ (the chance the answer is $1$ ) using odds: $odds = \frac{p}{1 - p}$ and we assume the model outputs the log of the odds: $lo g (\frac{p}{1 - p}) = z .$

WHY. Why odds and not $p$ itself? A probability is trapped in $[0, 1]$ , but odds run from $0$ (certain "no") up to $+ \infty$ (certain "yes"). Take the logarithm and the range explodes both ways: $- \infty$ to $+ \infty$ — exactly the range $z$ already lives in. So log-odds is the one transformation that makes "unbounded number" and "probability" speak the same language. Choosing $lo g$ (not some other stretch) is what makes feature effects add up linearly.

PICTURE. Three stacked bars showing the same belief in three languages: probability $p = 0.75$ (a bar capped at 1), odds $= 3$ (a taller bar), log-odds $\approx 1.1$ (a signed value on a two-way axis). The red arrows show each transform stretching the range wider.

Step 4 — Solve for $p$ : the sigmoid is forced into existence

WHAT. We now undo the log-odds to get $p$ back. Starting from $\frac{p}{1 - p} = e^{z}$ (exponentiate both sides — $e^{(\cdot)}$ cancels $lo g$ ): $p = e^{z} (1 - p) \Rightarrow p + p e^{z} = e^{z} \Rightarrow p (1 + e^{z}) = e^{z} \Rightarrow p = \frac{e ^{z}}{1 + e ^{z}} .$ Multiply top and bottom by $e^{- z}$ : $p = σ (z) = \frac{1}{1 + e ^{- z}}$

WHY. We didn't invent this S-curve for fun — it is the only function consistent with "log-odds are linear". Term by term: $e^{- z}$ shrinks to $0$ as $z \to + \infty$ (so $σ \to 1$ ), and blows up as $z \to - \infty$ (so $σ \to 0$ ). The $+ 1$ in the denominator guarantees we never divide by zero and never exceed $1$ .

PICTURE. The famous S: a smooth red curve from height $0$ (far left) to height $1$ (far right), passing dead-center through $(0, 0.5)$ . The flat tails and the steep middle are labelled.

All cases covered.

$z = 0 \Rightarrow σ = \frac{1}{1 + 1} = 0.5$ — the fence-sitting decision boundary.
$z \to + \infty \Rightarrow e^{- z} \to 0 \Rightarrow σ \to 1$ — total confidence in "yes".
$z \to - \infty \Rightarrow e^{- z} \to \infty \Rightarrow σ \to 0$ — total confidence in "no".

Step 5 — The slope of the S is a hidden gift

WHAT. How fast does the S climb? Differentiating (chain rule on $u = 1 + e^{- z}$ ): $σ^{'} (z) = \frac{e ^{- z}}{( 1 + e ^{- z} ) ^{2}} = σ (z) \frac{1}{1 + e ^{- z}} \cdot 1 - σ (z) \frac{e ^{- z}}{1 + e ^{- z}} = σ (z) (1 - σ (z)) .$

WHY. We need the slope because gradient descent moves downhill, and "downhill" means "in the direction of the slope". The magic is that the slope is expressed purely in terms of the output $σ$ itself — no leftover exponentials. That's why training is cheap: once you know the probability, you instantly know the slope.

PICTURE. The S-curve again in black, with its slope curve $σ (1 - σ)$ drawn in red beneath — a bell-shaped hump peaking at $z = 0$ (value $0.25$ ) and vanishing at both tails, showing where the curve is steep vs. flat.

Step 6 — Scoring surprise: cross-entropy

WHAT. We now need a number that says "how bad was this guess $\overset{y}{^} = σ (z)$ against the truth $y$ ?" The score is $L (y, \overset{y}{^}) = - [y lo g \overset{y}{^} + (1 - y) lo g (1 - \overset{y}{^})] .$

WHY. Look at how the two labels switch it on and off. Term by term:

If $y = 1$ : the $(1 - y)$ term vanishes, leaving $L = - lo g \overset{y}{^}$ .
If $y = 0$ : the $y$ term vanishes, leaving $L = - lo g (1 - \overset{y}{^})$ .

Only the term matching the true answer survives. And $- lo g (probability of the truth)$ is exactly "surprise": if you gave the truth probability $1$ , surprise is $- lo g 1 = 0$ ; if you gave it probability near $0$ , surprise rockets to $+ \infty$ . Confident wrong answers are punished mercilessly.

PICTURE. Two curves: red is $- lo g (\overset{y}{^})$ (the cost when $y = 1$ ) diving toward $0$ as $\overset{y}{^} \to 1$ and shooting up as $\overset{y}{^} \to 0$ ; black is $- lo g (1 - \overset{y}{^})$ (the cost when $y = 0$ ), its mirror image.

Step 7 — Why this exact score? Maximum likelihood

WHAT. Assume each label came from a coin biased with probability $\overset{y}{^}_{i}$ (a Bernoulli trial): $P (y_{i} ∣ \overset{y}{^}_{i}) = \overset{y}{^}_{i}^{y_{i}} (1 - \overset{y}{^}_{i})^{1 - y_{i}} .$ The chance of seeing all the data (independent) is the product; take its log and negate: $J = - \frac{1}{N} \sum_{i = 1}^{N} [y_{i} lo g \overset{y}{^}_{i} + (1 - y_{i}) lo g (1 - \overset{y}{^}_{i})] .$

WHY. We didn't pick cross-entropy by taste — it is the negative log-likelihood. Maximising likelihood = "choose weights that make the data we actually saw the least surprising". The $lo g$ turns an unwieldy product into a friendly sum (sums differentiate term-by-term); the minus turns "maximise" into "minimise", so we can hand it to gradient descent. This is the maximum-likelihood view.

PICTURE. A row of Bernoulli coins, each with its own bias $\overset{y}{^}_{i}$ , multiplied into one long product (red), then a downward arrow to the same thing rewritten as a sum of logs — visually shrinking a scary product into a tidy sum.

Step 8 — The miracle cancellation: the gradient

WHAT. We want the slope of the loss with respect to a weight $w_{j}$ . Chain three pieces: $\frac{\partial L}{\partial w _{j}} = - \frac{y}{y ^} + \frac{1 - y}{1 - y ^} \frac{\partial L}{\partial y ^} \cdot \overset{y}{^} (1 - \overset{y}{^}) \frac{\partial y ^}{\partial z} \cdot x_{j} \frac{\partial z}{\partial w _{j}} .$ Multiply the first two: the ugly fractions get cancelled by the $\overset{y}{^} (1 - \overset{y}{^})$ from Step 5: $[- y (1 - \overset{y}{^}) + (1 - y) \overset{y}{^}] = - y + y \overset{y}{^} + \overset{y}{^} - y \overset{y}{^} = \overset{y}{^} - y .$ So $\frac{\partial L}{\partial w _{j}} = (\overset{y}{^} - y) x_{j}$

WHY. This is why we chose sigmoid and cross-entropy together: the sigmoid's slope $\overset{y}{^} (1 - \overset{y}{^})$ (which caused saturation in Step 5) is exactly the thing that appears in the numerator of $\partial L / \partial \overset{y}{^}$ — and it cancels. The freeze is cured. What remains is pure meaning: error $(\overset{y}{^} - y)$ times feature $x_{j}$ . Wrong-and-confident? Big error, big correction.

PICTURE. A three-box chain diagram: box $\partial L / \partial \overset{y}{^}$ , box $\overset{y}{^} (1 - \overset{y}{^})$ , box $x_{j}$ feeding an arrow; the two middle terms shown crossing out in red, leaving $(\overset{y}{^} - y) x_{j}$ at the exit.

Degenerate cases.

Perfect prediction $\overset{y}{^} = y$ : gradient $= 0$ — nothing to fix, learning stops correctly.
$\overset{y}{^} \to 0$ or $1$ exactly: $lo g$ blows up numerically → clamp $\overset{y}{^}$ into $[ε, 1 - ε]$ in code (and see Regularization (L1, L2) to stop weights racing to $\pm \infty$ ).

The one-picture summary

WHAT. One figure that threads all eight steps: feature $x$ → linear score $z$ → sigmoid squashes into $\overset{y}{^}$ → compare to truth $y$ → cross-entropy surprise → slope collapses to $(\overset{y}{^} - y) x$ → nudge $w$ .

Recall Feynman retelling — say it like you're explaining to a friend

A machine can only add and multiply, so first it makes one plain number $z = w x + b$ . That number can be anything, but we wanted a probability, which has to sit between 0 and 1. The trick: assume $z$ is the log of the odds, because log-odds is the one thing that ranges over the whole number line just like $z$ does. Undo that assumption algebraically and out pops the S-shaped sigmoid — we didn't invent it, it was forced. The sigmoid's slope turns out to be $\overset{y}{^} (1 - \overset{y}{^})$ , written using only its own output. Then we need a report card: cross-entropy, which is just "how surprised am I to see the true label?" and it happens to be the negative log-likelihood of a coin-flip model, so it's the mathematically honest choice. Finally, when we ask how the report card changes as we tweak a weight, the sigmoid's slope cancels the messy fractions, and everything melts down to error times feature, $(\overset{y}{^} - y) x$ . That simplicity is the whole reason logistic regression is fast and stable.

Related destinations: Softmax Regression (many classes), Neural Networks (stack these units), ROC Curves and AUC (judge the classifier), Bayesian Interpretation (priors on $w$ ).