Logistic regression and the sigmoid function: Visual walkthrough

Step 1 — The problem: a straight line escapes the box

WHAT. We have a classification question with a yes/no answer. We want a probability — a number that must stay between $0$ and $1$ . Call it $p$ . Think of $p$ as "how sure we are the answer is yes". $p = 0$ means "certainly no", $p = 1$ means "certainly yes", $p = 0.5$ means "coin flip".

WHY. The only machine we know so far (Linear Regression) computes a straight-line score: $z = slope: how strongly a feature pushes w \cdot the input feature x + offset: the baseline score b .$ Here $w$ , $x$ , $b$ are just plain numbers for now. As $x$ slides from far-left to far-right, $z$ slides from $- \infty$ to $+ \infty$ . That's the trouble: a probability can't go below $0$ or above $1$ , but a straight line happily shoots past both.

PICTURE. The red line is the score $z$ . The grey band is the "legal zone" $[0, 1]$ where a probability must live. Watch the red line pierce straight through the top and bottom — it refuses to stay in the box.

Figure — Logistic regression and the sigmoid function

Step 2 — Odds: unlock the top of the box

WHAT. Instead of the probability $p$ directly, look at the odds: $odds = \frac{p}{1 - p} .$ Here $p$ is "chance of yes" and $1 - p$ is "chance of no". The ratio is "how many times more likely yes is than no". Gamblers say "3 to 1" — that's odds of $3$ .

WHY. This is our first stretch of the range. As $p$ climbs from $0$ toward $1$ :

at $p = 0$ , odds $= \frac{0}{1} = 0$ ,
at $p = 0.5$ , odds $= \frac{0.5}{0.5} = 1$ ,
as $p \to 1$ , the denominator $1 - p \to 0$ , so odds $\to + \infty$ .

So odds live in $(0, + \infty)$ . We unlocked the top of the box — the ceiling at $1$ is gone. But the floor is still stuck at $0$ : odds can never be negative. Half the job done.

PICTURE. The horizontal axis is $p$ (trapped in $[0, 1]$ ). The red curve is the odds. See it hug the left, cross through $1$ at $p = 0.5$ , then rocket to infinity as $p$ nears $1$ — the ceiling is broken, the floor is not.

Step 3 — Log-odds: unlock the bottom too

WHAT. Take the natural logarithm of the odds. This new quantity has a name — the logit: $logit (p) = ln odds, in (0, \infty) \frac{p}{1 - p} .$ The symbol $ln$ is the natural log: it answers "e raised to what power gives this number?" Its defining habit is turning a range that starts at $0$ into a range that starts at $- \infty$ .

WHY $ln$ and not something else? We need a function that maps $(0, \infty)$ onto the entire line $(- \infty, \infty)$ . The log does exactly that:

odds near $0$ (very unlikely) $\Rightarrow ln (tiny) \to - \infty$ (unlocks the floor),
odds $= 1$ (50/50) $\Rightarrow ln (1) = 0$ (the natural centre),
odds $\to \infty$ (very likely) $\Rightarrow ln (huge) \to + \infty$ (keeps the ceiling open).

No other simple function stretches a one-sided range onto the whole line so cleanly while keeping "50/50" pinned at $0$ . Now the log-odds range exactly matches the range of a straight-line score.

PICTURE. Same $p$ -axis in $[0, 1]$ . The red curve is the logit. Notice it dives to $- \infty$ on the left, passes through the origin at $p = 0.5$ , and climbs to $+ \infty$ on the right — both walls of the box are gone.

Step 4 — The bold equation: glue the two lines together

WHAT. Both sides now live on $(- \infty, \infty)$ , so we are allowed to set them equal: $log-odds of the answer ln \frac{p}{1 - p} = the straight-line score z w x + b .$

WHY. This is the one modelling assumption of logistic regression: the log-odds are a straight line in the input. That's all we assume. Everything else is algebra. It's a fair assumption to try first because a straight line is the simplest thing that can grow without bound — and log-odds need to grow without bound.

PICTURE. Two number lines, both running $- \infty \to + \infty$ , snapped together by an equals sign. The red bracket shows the join: "the score $z$ is the log-odds". Left of $z = 0$ means odds below 1 (leaning no); right of $z = 0$ means odds above 1 (leaning yes).

Step 5 — Unwrap to find $p$ : the exponential undoes the log

WHAT. We solve the boxed equation for $p$ . Apply $e^{(\cdot)}$ to both sides. The exponential $e^{u}$ is the exact inverse of $ln$ : $e^{l n u} = u$ . It undoes the logarithm, freeing the odds: $\frac{p}{1 - p} = e^{z} .$

WHY. We want $p$ alone, but $p$ is currently trapped inside a log and inside a fraction. Removing the log with $e$ is the first unlock. We use the exponential specifically because it is the one function that cancels $ln$ perfectly — no other tool reverses a logarithm.

Now the algebra, term by term:

p &= e^{z}\,(1-p) &&\text{multiply both sides by } (1-p)\\ p &= e^{z} - p\,e^{z} &&\text{distribute } e^{z}\\ p + p\,e^{z} &= e^{z} &&\text{collect every }p\text{ on the left}\\ p\,(1+e^{z}) &= e^{z} &&\text{factor out }p\\ p &= \frac{e^{z}}{1+e^{z}} &&\text{divide by }(1+e^{z}). \end{aligned}$$ One last polish — divide top and bottom by $e^{z}$ (allowed, $e^z\neq 0$): $$p = \frac{e^{z}/e^{z}}{(1+e^{z})/e^{z}} = \frac{1}{e^{-z}+1} = \frac{1}{1+e^{-z}} = \sigma(z).$$ Here each symbol: $\sigma$ is the name we give this squashing function; $z$ is the input score; $e^{-z}$ is "how fast the score fights back". When $z$ is big and positive, $e^{-z}$ is tiny, so $\sigma\approx 1$. **PICTURE.** A left-to-right "unwrapping" chain: $\ln\frac{p}{1-p}=z \Rightarrow \frac{p}{1-p}=e^z \Rightarrow p=\sigma(z)$. The red highlight follows $p$ as it gets freed at each arrow. ![[deepdives/dd-ai-ml-2.2.09-d2-s05.png]] --- ## Step 6 — Meet the S-curve, and read its whole shape **WHAT.** We've built it: $$\sigma(z) = \frac{1}{1+e^{-z}}.$$ This is the **sigmoid**. It is the translator we hunted for in Step 1: feed it any score on $(-\infty,\infty)$, it hands back a probability in $(0,1)$. **WHY it's shaped like an S.** Plug in the three landmark cases: - $z\to -\infty$: then $e^{-z}\to +\infty$, denominator huge, so $\sigma\to 0$. (Certain *no*.) - $z=0$: then $e^{0}=1$, so $\sigma=\frac{1}{1+1}=0.5$. (Perfect coin flip — the centre.) - $z\to +\infty$: then $e^{-z}\to 0$, denominator $\to 1$, so $\sigma\to 1$. (Certain *yes*.) Between these it rises smoothly, steepest at the middle and flattening at both ends — an "S". **PICTURE.** The red S-curve of $\sigma(z)$ over $z$. Dashed guides mark the floor at $0$, the ceiling at $1$, and the centre dot at $(0,0.5)$. Compare with Step 1: the same score $z$, but now it never leaves the box. ![[deepdives/dd-ai-ml-2.2.09-d2-s06.png]] > [!formula] The symmetry, and why it's free > $$\sigma(-z) = 1-\sigma(z).$$ > Flipping the sign of the score flips "yes" into "no". This is baked into the log-odds picture: negating $z$ negates the log-odds, which swaps $p$ with $1-p$. --- ## Step 7 — The degenerate & edge cases (never surprise the reader) **WHAT & WHY.** We check every corner so nothing can catch us off guard. - **$z=0$ (the boundary).** $\sigma(0)=0.5$. This is where "yes" and "no" tie. Setting $z=0$ i.e. $wx+b=0$ gives the [[Decision Boundaries|decision boundary]] — the exact input where the prediction flips. It's a single point here (one feature), a line or plane with more. - **$z\to +\infty$.** $\sigma\to 1$ but **never reaches** $1$. The interval is *open*: $(0,1)$, not $[0,1]$. Why it matters: the model is never *infinitely* certain, so $\ln p$ never blows up in the loss. - **$z\to -\infty$.** $\sigma\to 0$ but never reaches $0$, for the same reason. - **Slope $w=0$.** Then $z=b$ is constant, so $\sigma$ is a flat horizontal line — the model ignores the feature and predicts the same probability everywhere. - **Very large $|w|$.** The S-curve gets steep, nearly a vertical step: the model becomes almost a hard yes/no switch with little grey zone. **PICTURE.** Three sigmoids sharing the same $b$ but different slopes: a gentle one, a medium one, and a near-vertical red one for large $w$. All pass through the same centre; only the steepness changes. ![[deepdives/dd-ai-ml-2.2.09-d2-s07.png]] > [!mistake] "As $z$ grows the probability eventually equals 1." > **Why it feels right:** the curve visually flattens onto the line $1$. > **The fix:** $e^{-z}$ is always a *positive* number, so $1+e^{-z}>1$ always, so $\sigma(z)<1$ always. It gets *arbitrarily close* to $1$ but the output stays strictly inside $(0,1)$ forever. --- ## The one-picture summary Everything in one frame: on the **left**, probability $p$ trapped in $[0,1]$; the two stretching moves (odds → log-odds) blow the box open onto the full line; the straight-line score $z=wx+b$ lands there; then the sigmoid folds it all back into $(0,1)$ on the **right**. The red loop is the round trip — $p \to z \to p$ — that logistic regression rides. ![[deepdives/dd-ai-ml-2.2.09-d2-s08.png]] > [!recall]- Feynman retelling — the whole walkthrough in plain words > We wanted a confidence number that stays between 0 and 1, but our only tool draws straight lines that run off to infinity in both directions. So we played a stretching game. First we swapped the probability for *odds* ("how many times more likely yes than no") — that snapped off the ceiling, letting the number climb to infinity. Then we took the *log* of the odds — that snapped off the floor too, letting it dive to minus-infinity. Now our confidence-measure lives on the same endless line as a straight-line score, so we boldly declared them equal: **the score is the log-odds.** To read the confidence back out, we simply reversed the two stretches — the exponential undid the log, a pinch of algebra undid the odds — and out popped the S-shaped sigmoid $\frac{1}{1+e^{-z}}$. It squashes any score into a probability: minus-infinity gives 0, zero gives a 50/50, plus-infinity gives 1, and it never quite touches either wall. That gentle S is not a random choice — it is the *only* curve you get once you insist the log-odds be a straight line. > [!mnemonic] The stretch-and-squash chant > **"Odds break the ceiling, logs break the floor, the line lands, the sigmoid squashes it back."** ## Connections - [[Linear Regression]] — where the straight-line score $z=wx+b$ comes from. - [[Decision Boundaries]] — the point/line $z=0$ from Step 7. - [[Maximum Likelihood Estimation]] & [[Cross-Entropy Loss]] — how we *choose* $w,b$ (next page). - [[Gradient Descent]] — the engine that nudges the weights. - [[Softmax Regression]] — the same idea with more than two classes.