Exercises — Decision boundaries

WHAT: set the score to zero. WHY: border ⇔ $z=0$. $3x-6=0\Rightarrow x=2.$ For $x>2$: $z>0\Rightarrow \sigma (z)>0.5\Rightarrow$ class 1. For $x<2$: class 0. Answer: boundary at $==x=2==$; class 1 lies to the right.

Linear means every feature appears to the first power, no products, no squares. (a) linear ✅ — a straight line. (b) has squares → circle. (c) has a product $x_1{x}_2$ → hyperbola. Answer: only (a).

Boundary: $z=0\Rightarrow 2x_1+x_2=4$. Intercepts: set $x_2=0\Rightarrow x_1=2$, giving $(2,0)$. Set $x_1=0\Rightarrow x_2=4$, giving $(0,4)$. Classify $(1,1)$: $z=2(1)+1-4=-1<0\Rightarrow$ class 0. Classify $(3,0)$: $z=2(3)+0-4=2>0\Rightarrow$ class 1. See the figure below — both points sit on opposite sides of the amber line.

WHY use the length: raw $z$ is stretched by how big $\mathbf{w}$ is; dividing by $\Vert \mathbf{w}\Vert$ gives true units. $z=3(4)+4(2)-10=12+8-10=10.$ $\Vert \mathbf{w}\Vert =\sqrt{3^2+4^2}=\sqrt{25}=5.$ $d=\frac{10}{5}=2.$ Positive ⇒ class 1, and it lies exactly 2 units on the class-1 side.

WHAT: a threshold $t$ means boundary at $\sigma (z)=t$, i.e. $z=\ln \frac{t}{1-t}$. $z=\ln \frac{0.8}{0.2}=\ln 4\approx \mathrm{1.3863.}$ So $x_1+x_2-2=\ln 4\Rightarrow x_1+x_2=2+\ln 4\approx \mathrm{3.3863.}$ Orientation: the coefficients of $x_1,x_2$ (i.e. $\mathbf{w}=(1,1)$) are unchanged, so the line only slides parallel to itself; its tilt is identical. Raising the threshold pushes the border toward the class-1 region (need more evidence to call "1").

Boundary of A: $x_1+2x_2-3=0$. Boundary of B: $5x_1+10x_2-15=0$. Divide every term by $5$: $x_1+2x_2-3=0.$ Identical set of points ⇒ same boundary. Since $z_B=5z_A$, wherever $z_A=0$ we also have $z_B=0$; the factor $5$ never changes the zero-set — it only makes the sigmoid steeper (higher confidence away from the line).

WHY squared features: a plain line can never bend; but the same linear rule on new features $x_1^2,x_2^2$ traces a curve when we plot back in $(x_1,x_2)$ (see Feature Engineering). Want boundary $x_1^2+x_2^2=9$, i.e. $x_1^2+x_2^2-9=0$. So $z=x_1^2+x_2^2-9,w_0=-9,w_3=1,w_4=1.$ Classify $(1,1)$: $z=1+1-9=-7<0\Rightarrow$ class 0 (inside the circle). Classify $(4,0)$: $z=16+0-9=7>0\Rightarrow$ class 1 (outside).

WHAT: the boundary between two classes is where their scores tie: $z_A=z_B$. $x-1=3-x\Rightarrow 2x=4\Rightarrow x=2.$ For $x>2$: $z_A>z_B\Rightarrow$ class A. For $x<2$: class B. The tie generalises "$z=0$" — with several classes the border is where two winning scores are equal, not where one score is zero.

Step 1 — line direction fixes $\mathbf{w}$'s direction. The line through $(0,2)$ and $(4,0)$ has equation $\frac{x_1}{4}+\frac{x_2}{2}=1\Rightarrow x_1+2x_2-4=0$. So $\mathbf{w}\propto (1,2)$ and $b\propto -4$; write $\mathbf{w}=(c,2c),b=-4c$. Step 2 — the distance fixes the scale $c$. Signed distance of origin: $d(0,0)=\frac{c\cdot 0+2c\cdot 0-4c}{\sqrt{c^2+4c^2}}=\frac{-4c}{\sqrt{5}|c|}.$ For $c>0$ this is $\frac{-4c}{\sqrt{5}c}=\frac{-4}{\sqrt{5}}\approx -1.789$ — wrong magnitude. We need $d=-1$, so rescale by the direction only. Set $\mathbf{w}=(a,2a)$, $b=-4a$ and require length so distance $=-1$: $\frac{-4a}{\sqrt{5}a}=-1\Rightarrow \frac{-4}{\sqrt{5}}=-1?$ That is fixed, so instead we must relax $b$'s tie to the same scale is forced — the boundary line is fixed, so $d(0,0)$ is a fixed geometric number $=-4/\sqrt{5}$. Therefore a signed distance of exactly $-1$ for the origin is impossible for this line. Mastery insight: the boundary line already pins the origin's true distance at $-4/\sqrt{5}\approx -1.789$; you cannot choose it. Sign ($-$, class 0) is achievable, magnitude is not. Answer: no such $(\mathbf{w},b)$ exists; the consistent facts are $\mathbf{w}=(1,2),b=-4$ giving $d(0,0)=-4/\sqrt{5}$.

Boundary at threshold $t$: $\sigma (z)=t$. Invert the monotone sigmoid: $\frac{1}{1+e^{-z}}=t\Rightarrow 1+e^{-z}=\frac{1}{t}\Rightarrow e^{-z}=\frac{1-t}{t}\Rightarrow z=\ln \frac{t}{1-t}.$ So ${\mathbf{w}}^{\top }\mathbf{x}+b=\ln \frac{t}{1-t}$, i.e. ${\mathbf{w}}^{\top }\mathbf{x}+(b-\ln \frac{t}{1-t})=0$. The coefficient vector $\mathbf{w}$ is untouched ⇒ same normal, same orientation; only the offset changed by $-\ln \frac{t}{1-t}$. At $t=0.5$, $\ln \frac{0.5}{0.5}=\ln 1=0$, recovering $z=0$. ∎