Noesis
2.2.11Linear & Logistic Regression

Decision boundaries

1,648 words7 min readdifficulty · medium1 backlinks
🇮🇳 Yeh note Hinglish mein padho →

WHAT is a decision boundary?

WHY 0.5? Because for two classes the probabilities sum to 1. They are equal exactly when each is 0.50.5. On one side P(y=1)>0.5P(y=1)>0.5 (predict 1), on the other P(y=1)<0.5P(y=1)<0.5 (predict 0). The knife-edge is the boundary.

HOW to derive the boundary for Logistic Regression

Logistic regression models P(y=1∣x)=σ(z),z=w⊤x+b,σ(z)=11+e−z.P(y=1\mid \mathbf{x}) = \sigma(z), \qquad z = \mathbf{w}^\top \mathbf{x} + b, \qquad \sigma(z)=\frac{1}{1+e^{-z}}.

Step 1 — Set the boundary condition. We want P(y=1∣x)=0.5P(y=1\mid\mathbf{x})=0.5. Why this step? By definition the boundary is where the two classes tie, and the tie sits at 0.50.5.

Step 2 — Invert the sigmoid. σ(z)=12  ⟹  11+e−z=12  ⟹  1+e−z=2  ⟹  e−z=1  ⟹  z=0.\sigma(z)=\tfrac12 \;\Longrightarrow\; \frac{1}{1+e^{-z}}=\tfrac12 \;\Longrightarrow\; 1+e^{-z}=2 \;\Longrightarrow\; e^{-z}=1 \;\Longrightarrow\; z=0. Why this step? The sigmoid is monotonic, so its value 0.5 corresponds to exactly one input, z=0z=0. This converts a probability condition into a clean linear condition.

Step 3 — Write the boundary equation.   w⊤x+b=0  \boxed{\;\mathbf{w}^\top \mathbf{x} + b = 0\;} Why this step? z=0z=0 is the boundary. This is the equation of a hyperplane (a line in 2D, a plane in 3D).

WHY linear? Because σ\sigma only reshapes a linear score zz into a probability. The shape of the boundary is decided entirely by where z=0z=0, and zz is a straight-line function of x\mathbf{x}. The sigmoid bends the probabilities, not the boundary.

Figure — Decision boundaries

Making the boundary curved

If the true separation is curved, feed nonlinear features into the same linear machinery. Example: add x12,x22,x1x2x_1^2, x_2^2, x_1x_2. Then z=w0+w1x1+w2x2+w3x12+w4x22+w5x1x2=0z = w_0 + w_1x_1 + w_2x_2 + w_3x_1^2 + w_4x_2^2 + w_5x_1x_2 = 0 is linear in the weights but a conic (circle/ellipse/curve) in the original (x1,x2)(x_1,x_2) space.

Worked examples

Common mistakes

Recall Feynman: explain to a 12-year-old

Imagine drawing a line on the playground with chalk. Kids on the left are "Team Red", kids on the right are "Team Blue". The chalk line is the decision boundary — it's the fence deciding which team you're on. The computer's "how sure am I?" meter reads 50/50 exactly on the line, and gets more confident the farther you walk away from it. If a straight chalk line can't separate the teams (say Reds are in a ring around Blues), we let the computer draw a circle instead by looking at fancier clues like "distance from the center."

Active recall

What condition defines a two-class decision boundary in probability terms?
P(y=1∣x)=0.5P(y=1\mid\mathbf{x})=0.5 (the two classes tie).
For logistic regression, what equation is the decision boundary?
w⊤x+b=0\mathbf{w}^\top\mathbf{x}+b=0 (a hyperplane).
Why does P=0.5P=0.5 reduce to z=0z=0?
Because σ(z)=0.5\sigma(z)=0.5 only when z=0z=0 (sigmoid is monotonic).
Is the logistic regression boundary curved?
No — it is a linear hyperplane; only the probability surface is curved.
How do you get a curved boundary with a linear model?
Add nonlinear features (e.g. x2x^2, x1x2x_1x_2); boundary is linear in feature space, curved in input space.
What does the vector w\mathbf{w} represent geometrically?
The normal to the boundary, pointing toward the class-1 region.
Does scaling w\mathbf{w} and bb by c>0c>0 move the boundary?
No — the set w⊤x+b=0\mathbf{w}^\top\mathbf{x}+b=0 is unchanged; only confidence steepness changes.
What does changing the classification threshold do to the boundary?
Shifts the hyperplane parallel to itself (changes offset bb), keeps orientation.
Classify (1,1)(1,1) for z=x1+x2−3z=x_1+x_2-3.
z=−1<0z=-1<0 → class 0.
Boundary of z=x12+x22−4z=x_1^2+x_2^2-4?
Circle of radius 2 centred at origin.

Connections

  • Logistic Regression — where the z=0z=0 boundary comes from.
  • Sigmoid Function — monotonicity is why 0.5 ⇔ z=0z=0.
  • Linear Regression — same linear score zz, different output/loss.
  • Feature Engineering — turning linear boundaries into curved ones.
  • Support Vector Machines — maximum-margin version of the same hyperplane idea.
  • Softmax Classification — multi-class generalisation with several boundaries.

Concept Map

splits

defined where

because

models

set to 0.5

yields

is a

normal vector

sign of distance

make

still linear in

Decision boundary

Feature space into class regions

P y=1 equals P y=0 equals 0.5

Two probabilities sum to 1

Logistic regression

P y=1 equals sigmoid of z

Invert sigmoid gives z=0

w dot x plus b equals 0

Hyperplane, linear boundary

w points to class-1 side

Predicted class

Nonlinear features x1^2 x1x2

Curved conic boundary

Weight space

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, decision boundary ka matlab simple hai: ye wo "line" (ya surface) hai jahan model exactly confuse ho jaata hai — dono class equally likely lagti hain, yaani P(y=1)=0.5P(y=1)=0.5. Ek taraf model bolega "class 1", doosri taraf "class 0". Ye boundary hi decide karti hai ki tumhara model kaunsa shape separate kar sakta hai.

Logistic regression mein boundary nikaalna easy hai. Probability output σ(z)\sigma(z) hai, jahan z=w⊤x+bz=\mathbf{w}^\top\mathbf{x}+b. Boundary pe P=0.5P=0.5, aur sigmoid 0.50.5 tabhi deta hai jab z=0z=0. Toh boundary ka equation ban jaata hai w⊤x+b=0\mathbf{w}^\top\mathbf{x}+b=0 — ye ek seedhi line (ya hyperplane) hai. Yaad rakho: "Zero is the border." Sigmoid curve dekh kar mat socho ki boundary bhi curved hai — sigmoid sirf probability ko bend karta hai, boundary flat hi rehti hai.

Agar data curved separation maangta hai (jaise ek class doosri ke aas-paas circle mein), toh ghabrao mat — bas nonlinear features add kar do jaise x12,x22x_1^2, x_2^2. Tab z=x12+x22−4=0z=x_1^2+x_2^2-4=0 ek circle ban jaata hai. Trick ye hai: boundary hamesha feature space mein linear hoti hai, bas original input mein curved dikhti hai.

Ye concept isliye important hai kyunki interviews aur exams mein log yahin galti karte hain — sochte hain logistic regression ka boundary curved hota hai. Sahi jawab: linear, unless tum features nonlinear banao. Iss ek clarity se tumhara aadha chapter clear ho jaata hai (80/20 rule).

Go deeper — visual, from zero

D1FoundationsEvery symbol from zero · 1,757 wordsD2Visual walkthroughThe derivation in pictures · 2,102 wordsD3Worked examplesEvery scenario covered · 2,941 wordsD4ExercisesGraded, full solutions · 1,842 wordsD5Question bankConcept traps, quick reveals · 1,331 words

Test yourself — Linear & Logistic Regression

L1 RecognitionL2 RecallL3 ProductionL4 ApplicationL5 Mastery

Connections

Logistic regression — sigmoid, cross-entropy lossCoding · 5.6.2Logistic regression and the sigmoid functionAI-ML · 2.2.9Linear regression — normal equation, gradient descent derivationCoding · 5.6.1Logistic regression and the sigmoid functionAI-ML · 2.2.9