Foundations — Decision boundaries

Before you can derive that fence (the job of Decision boundaries), you must be fluent in every symbol the parent note throws at you. This page builds each one from nothing — plain words, a picture, and why the topic can't live without it.

0. The stage: feature space

Everything happens on a map. Each thing we want to classify (an email, a patient, a flower) is described by numbers called features. Two features = a 2D map, three = a 3D room, and so on.

Why the topic needs it: A decision boundary is a shape drawn on this map — you cannot talk about a boundary without first agreeing on the map it splits.

1. The symbols, in build order

1.1 $x$, $x_1$, $x_2$ — a single feature (a number)

Plain words: a lowercase $x$ with a little number below is one measurement — one axis of the map. $x_1$ might be "height", $x_2$ might be "weight".

Picture: a position along one axis. $x_1=3$ means "walk 3 steps along the first axis".

Why needed: the boundary equation is built out of these individual numbers.

1.2 $\mathbf{x}$ — a vector (a whole point, all features at once)

Why needed: the boundary condition ${\mathbf{w}}^{\top }\mathbf{x}+b=0$ talks about a whole point at once — so we need one letter for "the whole point".

1.3 $\mathbf{w}$ — the weight vector (the model's opinion)

Plain words: $\mathbf{w}=(w_1,w_2,\dots )$ is a list of importance dials, one per feature. A big $w_1$ means "feature 1 matters a lot for deciding the class".

Picture: an arrow living on the same map as $\mathbf{x}$. As we'll see, this arrow points perpendicular to the fence.

Why needed: $\mathbf{w}$ decides the tilt (orientation) of the boundary.

1.4 $b$ — the bias (a shift number)

Plain words: a single number added on top, the offset. It slides the fence back and forth without tilting it.

Picture: pushing the whole chalk line sideways, keeping its angle fixed.

Why needed: without $b$ the fence would be forced through the origin $(0,0)$; $b$ frees it to sit anywhere.

1.5 ${\mathbf{w}}^{\top }\mathbf{x}$ — the dot product (combining opinion with the point)

This is the workhorse. Let's earn it slowly.

WHY the dot product and not, say, just adding features? Because we need each feature to carry different weight — a straight average can't do that. The dot product is the simplest tool that lets every feature vote with its own strength. Geometrically it also measures how far along the $\mathbf{w}$ direction a point sits, which is exactly the "which side of the fence" question.

1.6 $z$ — the linear score

Plain words: stack the weighted sum and the shift together: $z={\mathbf{w}}^{\top }\mathbf{x}+b.$ This single number $z$ is the raw score: positive leans class 1, negative leans class 0, zero is the tie.

Picture: a signed "how far past the fence" reading. On the fence $z=0$; walk toward class 1 and $z$ climbs positive; walk the other way and it dives negative.

Why needed: the entire topic collapses to solving $z=0$. Everything else is dressing.

1.7 $\sigma (z)$ — the sigmoid (score → probability)

Here $e\approx 2.718$ is a fixed constant (the natural number) and $e^{-z}$ means "$e$ raised to the power $-z$". You don't need to compute it by hand — you need to know the shape.

WHY a sigmoid and not just $z$? Because a probability must live between 0 and 1, but $z$ can be any number (like $47$ or $-1000$). The sigmoid is the tool that bends an unbounded score into a legal probability, while never changing the order (bigger $z$ → bigger probability).

1.8 $P(y=1\mid \mathbf{x})$ — a conditional probability

Plain words: "the chance that the label $y$ is class 1, given the point $\mathbf{x}$". The bar $\mid$ reads "given / knowing".

Picture: a confidence dial reading between 0 and 1 attached to every dot on the map.

Why needed: the boundary is defined as where this dial reads $0.5$.

1.9 The threshold $0.5$ and the boundary condition

Two classes, and their probabilities must sum to 1: $P(y=1\mid \mathbf{x})+P(y=0\mid \mathbf{x})=1.$ They are equal exactly when each is $0.5$. That knife-edge tie is the boundary.

1.10 Hyperplane and normal vector

Why needed: knowing $\mathbf{w}$ is perpendicular is what lets us measure distance from the fence and read off the class from a sign.

1.11 $\Vert \mathbf{w}\Vert$ — the length of the weight vector

The double bars mean "length of the arrow": $\Vert \mathbf{w}\Vert =\sqrt{w_1^2+w_2^2+\dots }.$ It appears in the distance-from-boundary formula $\frac{{\mathbf{w}}^{\top }\mathbf{x}+b}{\Vert \mathbf{w}\Vert }$, where dividing by the length turns the raw score into an honest distance in map-units.

2. Why scaling $\mathbf{w},b$ doesn't move the fence (a symbol check)

If you multiply both $\mathbf{w}$ and $b$ by any positive $c$: $c{\mathbf{w}}^{\top }\mathbf{x}+cb=c({\mathbf{w}}^{\top }\mathbf{x}+b)=0⟺{\mathbf{w}}^{\top }\mathbf{x}+b=0.$ The set of solution points is identical. This is why the parent note insists magnitude changes confidence steepness (how fast the sigmoid saturates), not the fence location. Now you can see it's pure algebra on the symbols you just learned.

3. Prerequisite map

Equipment checklist

Test yourself — cover the right side and answer out loud.

Connections

• Decision boundaries — the parent topic these foundations feed into.
• Sigmoid Function — the S-curve and its monotonicity, built here.
• Logistic Regression — assembles $z$, $\sigma$, and the $z=0$ fence.
• Linear Regression — same linear score $z$, different output.
• Feature Engineering — where new features (like $x_1^2$) come from to curve the fence.
• Support Vector Machines — leans hard on the normal vector and margin.
• Softmax Classification — many probabilities and many fences at once.