Decision boundaries: Visual walkthrough — AI-ML deep dive

Step 1 — What "feature space" even is

WHAT. Imagine a flat sheet of paper. Every point on it is one thing we want to classify — say, one animal described by two measurements: its weight ( $x_{1}$ , left–right) and its height ( $x_{2}$ , up–down). One dot = one animal.

WHY. Before we can talk about a boundary, we need the place the boundary lives in. That place is the sheet of paper: the feature space. A classifier's whole job is to colour this sheet into regions — one colour per class.

PICTURE. Look at the figure. The horizontal axis is $x_{1}$ , the vertical axis is $x_{2}$ . Blue dots are class 1 (say "dogs"), orange dots are class 0 ("cats"). Right now there is no line — just scattered points. Our mission: find the fence.

Step 2 — The score $z$ : turning a point into a single number

WHAT. We invent a rule that eats a point $x = (x_{1}, x_{2})$ and spits out one number called the score: $z = tilt in x_{1} w_{1} x_{1} + tilt in x_{2} w_{2} x_{2} + shift b .$

$x_{1}, x_{2}$ — the point's coordinates (its two features).
$w_{1}, w_{2}$ — the weights: how strongly each feature pushes the score up or down.
$b$ — the bias: a constant nudge, up or down, independent of the point.
$z$ — the resulting score, a single real number, positive or negative.

WHY this tool and not another? We want the simplest recipe that combines the features. A weighted sum $w_{1} x_{1} + w_{2} x_{2}$ is the simplest — nothing multiplied together, no squares, just "add up scaled features." We choose it precisely because it is linear; that simplicity is what forces the boundary to be straight (we prove this in Step 6).

PICTURE. Think of $z$ as a height above the sheet. The recipe $z = w_{1} x_{1} + w_{2} x_{2} + b$ is a tilted flat ramp floating over the paper. Every point on the sheet gets lifted to a height. The figure shows this ramp as a tilted plane, with a few points and their heights marked.

Step 3 — The sigmoid: squashing the score into a probability

WHAT. The score $z$ can be any number from $- \infty$ to $+ \infty$ . But a probability must live between 0 and 1. So we pass $z$ through a squasher, the sigmoid: $σ (z) = \frac{1}{1 + e ^{- z}} .$

$e$ — the number $\approx 2.718$ ; $e^{- z}$ is " $e$ raised to minus the score."
When $z$ is very negative, $e^{- z}$ is huge, so $σ \to 0$ .
When $z$ is very positive, $e^{- z} \to 0$ , so $σ \to 1$ .
When $z = 0$ , $e^{- 0} = 1$ , so $σ = \frac{1}{1 + 1} = \frac{1}{2}$ .

WHY this tool? We need a function that (a) always outputs between 0 and 1, and (b) is monotonic — always increasing, never turning back. Monotonic is the magic word: it means every output value comes from exactly one input. That one-to-one property is what lets us "undo" it cleanly in Step 5. See the Sigmoid Function note for its full story.

PICTURE. The figure is the famous S-curve. Read it left to right: floor at 0 on the far left, ceiling at 1 on the far right, and it passes through exactly the point $(0, 0.5)$ — marked with a red dot. That red dot is the hero of this entire page.

Step 4 — The boundary is the 50/50 tie

WHAT. The classifier decides: predict class 1 if $σ (z) > 0.5$ , else class 0. The decision boundary is the knife-edge in between, where it is perfectly torn: $P (y = 1 ∣ x) = 0.5.$

WHY $0.5$ ? The two class probabilities add to 1. They are equal only when each is exactly $\frac{1}{2}$ . That is the moment of maximum indecision — the fence.

PICTURE. Same S-curve as before, but now we slice it horizontally at height $0.5$ (a dashed red line). Where the slice meets the curve is the red dot — and it sits directly above $z = 0$ . Above the slice → predict 1 (blue shading). Below → predict 0 (orange shading).

Step 5 — Undoing the sigmoid: why $0.5$ means $z = 0$

WHAT. We solve $σ (z) = \frac{1}{2}$ for $z$ , step by algebra step:

\;\Longrightarrow\; 1+e^{-z} = 2 \;\Longrightarrow\; e^{-z} = 1 \;\Longrightarrow\; z = 0.$$ - **Cross-multiply:** $\frac{1}{1+e^{-z}}=\frac12$ means $1+e^{-z}$ must equal $2$. - **Subtract 1:** $e^{-z} = 1$. - **What power of $e$ gives 1?** Only the power $0$, since $e^0 = 1$. So $-z = 0$, i.e. $z = 0$. **WHY it is clean (one answer, not many).** Because the sigmoid is monotonic (Step 3), the value $0.5$ is hit at **exactly one** $z$. No ambiguity. This is the payoff of choosing a monotonic squasher — we get a single, tidy condition. **PICTURE.** The figure shows the arrow going *backwards* through the S-curve: start at height $0.5$ on the vertical axis, travel left to the curve, drop straight down — you land at $z=0$ and nowhere else. ![[deepdives/dd-ai-ml-2.2.11-d2-s05.png]] > [!mnemonic] Zero is the border > The whole probability question collapses to one line: **set $z=0$**. Everything after is algebra. --- ## Step 6 — $z=0$ drawn on the sheet: a straight fence **WHAT.** Substitute the recipe for $z$ into $z=0$: $$\boxed{\;w_1 x_1 + w_2 x_2 + b = 0\;}$$ This is the equation of a **straight line** on our sheet of paper (a ==hyperplane== in higher dimensions). - Rearranged: $x_2 = -\dfrac{w_1}{w_2}x_1 - \dfrac{b}{w_2}$ — a line with slope $-w_1/w_2$ and intercept $-b/w_2$. **WHY it is straight (the key insight).** The score $z$ is a *tilted flat ramp* (Step 2). Setting $z=0$ asks: *where does the ramp cross the floor?* A flat ramp crosses a flat floor along a **straight line** — always. The sigmoid bent the *heights*, but it never touched *where* $z=0$. So the fence is straight no matter how curvy the S-curve looked. **PICTURE.** Take the tilted ramp of Step 2 and mark the level line where its height is exactly zero. Projected down onto the sheet, that level line is our straight boundary. Blue on the $z>0$ side, orange on the $z<0$ side. ![[deepdives/dd-ai-ml-2.2.11-d2-s06.png]] > [!mistake] "The sigmoid curves, so the boundary must curve." > The S-curve lives in *probability* space (height). The boundary lives in *feature* space (the floor). Slicing a curved surface at one height can still give a straight cut — and here it always does, because $z$ is flat. See [[Feature Engineering]] to make the boundary genuinely curved. --- ## Step 7 — Which side is which? The normal vector $\mathbf{w}$ **WHAT.** The pair $\mathbf{w} = (w_1, w_2)$, drawn as an arrow, points **perpendicular** to the boundary, toward the class-1 side. The signed distance of any point $\mathbf{x}$ from the fence is $$\text{distance} = \frac{w_1 x_1 + w_2 x_2 + b}{\sqrt{w_1^2 + w_2^2}} = \frac{z}{\lVert\mathbf{w}\rVert}.$$ - Numerator is just $z$ — the score. - Denominator $\lVert\mathbf{w}\rVert=\sqrt{w_1^2+w_2^2}$ is the arrow's length, used to convert "score units" into "real distance." - **Sign** of $z$ = sign of distance = which side = which class. **WHY.** Walking in the direction of $\mathbf{w}$ increases $z$ fastest (it is the direction of steepest uphill on the ramp). So $\mathbf{w}$ literally arrows toward "more class 1." Farther from the fence → bigger $|z|$ → more confident. **PICTURE.** The straight boundary, with the orange arrow $\mathbf{w}$ stabbing across it at a right angle into the blue region. A point on the blue side has its perpendicular drop to the line measured; that length times $\lVert\mathbf{w}\rVert$ is the raw score. ![[deepdives/dd-ai-ml-2.2.11-d2-s07.png]] --- ## Step 8 — Degenerate & edge cases (the fence still behaves) **WHAT & WHY & PICTURE**, three quick corners the reader might hit: 1. **$w_2 = 0$ (vertical fence).** Then $z = w_1 x_1 + b$, and $z=0$ gives $x_1 = -b/w_1$ — a **vertical** line. The slope formula $-w_1/w_2$ divides by zero; that "infinity" is not a bug, it *is* vertical. (Panel A.) 2. **$w_1 = w_2 = 0$ (no fence at all).** Then $z = b$, a constant everywhere. If $b\neq0$ every point gets the same class — no boundary exists, the whole sheet is one colour. (Panel B.) 3. **Change the threshold $t$ instead of $0.5$.** Boundary becomes $\sigma(z)=t \Rightarrow z = \ln\frac{t}{1-t}=$ a constant $c$. So $w_1x_1+w_2x_2+b=c$: the *same-direction* line, just **slid parallel** to itself. Orientation ($\mathbf{w}$) is untouched. (Panel C.) 4. **Scale $\mathbf{w},b$ by $c>0$.** The set $\{c\,z = 0\}$ is identical to $\{z=0\}$ — the fence does **not** move. Only the S-curve gets steeper (more confidence). (Panel D.) ![[deepdives/dd-ai-ml-2.2.11-d2-s08.png]] > [!example] Numbers to hold onto > For $z = x_1 + x_2 - 3$: boundary $x_1+x_2=3$. Point $(1,1)$ gives $z=-1<0 \Rightarrow$ class 0. Point $(3,3)$ gives $z=3>0\Rightarrow$ class 1. --- ## The one-picture summary Everything on one canvas: the tilted ramp $z$ (Step 2) casts its **zero level line** (Step 6) onto the sheet as a straight fence; the sigmoid (Step 3) reads $0.5$ (Step 4) exactly over $z=0$ (Step 5); and the arrow $\mathbf{w}$ (Step 7) points to the blue side. ![[deepdives/dd-ai-ml-2.2.11-d2-s09.png]] > [!recall]- Feynman retelling — the whole walk in plain words > Draw a sheet of paper; every dot is one animal described by two numbers. We build a simple machine: multiply each number by a weight, add them, add a fixed nudge — that gives one score $z$. Picture $z$ as a tilted flat ramp hovering over the paper. Now squash $z$ with an S-shaped curve so it becomes a "how sure am I it's a dog" between 0 and 1. The fence — the decision boundary — is where we're perfectly unsure, 50/50. On the S-curve, 50/50 sits exactly over score zero, and because the S-curve never doubles back, that's the *only* place. So the fence is simply "where the ramp's height is zero." A flat ramp crosses the flat floor along a **straight** line — that's why logistic regression always draws a straight fence, no matter how curvy the S-curve looked. The weight-arrow points across the fence toward the dog side, and the farther you stand from the fence, the surer the machine is. > [!recall]- > Why does $P=0.5$ become $z=0$? ::: Because the sigmoid is monotonic, so $0.5$ is reached at exactly one input, $z=0$. > Why is the boundary straight? ::: $z$ is a flat (linear) ramp; where a flat ramp hits height zero is a straight line. > What does $\mathbf{w}$ point toward? ::: Perpendicular to the fence, into the class-1 region. > Does scaling $\mathbf{w},b$ by $c>0$ move the fence? ::: No — only the sigmoid's steepness (confidence) changes. > What happens with a different threshold $t$? ::: The fence slides parallel to itself ($z=\ln\frac{t}{1-t}$); orientation stays. --- ## Connections - [[Decision boundaries|Parent: Decision boundaries]] — this page is its visual derivation. - [[Logistic Regression]] — the model whose boundary we drew. - [[Sigmoid Function]] — monotonicity is why $0.5\Leftrightarrow z=0$. - [[Linear Regression]] — same linear score $z$, different output. - [[Feature Engineering]] — bend the straight fence into a curve. - [[Support Vector Machines]] — the maximum-margin cousin of this hyperplane. - [[Softmax Classification]] — several of these boundaries at once for many classes.