Logistic regression and the sigmoid function

2.2.9 · AI-ML › Linear & Logistic Regression

WHAT hai logistic regression?

Figure — Logistic regression and the sigmoid function

WHY specifically sigmoid? (Odds se derivation)

Hume ek aisi function chahiye jo linear score ko probability mein badal sake. Odds se shuru karo, probability se nahi.

Formula Derivation: log-odds → sigmoid

Step 1. Maano $p = P (y = 1 ∣ x)$ . Odds hain $\frac{p}{1 - p} \in (0, \infty)$ . Yeh step kyun? Probability $[0, 1]$ mein rehti hai (bounded), lekin ek linear model kuch bhi output kar sakta hai. Odds $(0, \infty)$ mein rehte hain — thoda karib, par phir bhi ek-taraf.

Step 2. Log-odds lo (jise logit kehte hain): $ln \frac{p}{1 - p} \in (- \infty, \infty)$ . Yeh step kyun? Ab range bilkul ek linear function ki range se match karti hai. Toh hum dilerri se set kar sakte hain: $ln \frac{p}{1 - p} = w^{⊤} x + b = z .$

Step 3. $p$ ke liye solve karo. Exponentiate karo: $\frac{p}{1 - p} = e^{z}$ . Kyun? Log ko undo karo taaki $p$ alag ho jaye.

Step 4. $p = e^{z} (1 - p) \Rightarrow p + p e^{z} = e^{z} \Rightarrow p (1 + e^{z}) = e^{z}$ $p = \frac{e ^{z}}{1 + e ^{z}} = \frac{1}{1 + e ^{- z}} = σ (z) .$ Kyun? Upar aur neeche dono ko $e^{z}$ se divide karo taaki clean canonical form mile.

Toh sigmoid arbitrary nahi hai — yeh log-odds ka inverse hai. Yeh assume karna ki log-odds $x$ mein linear hain, sigmoid ko force karta hai.

HOW train karte hain? (Loss aur uska gradient)

Hum squared error easily use nahi kar sakte (sigmoid ke liye yeh non-convex hai). Iski jagah maximum likelihood use karo.

Decision boundary

Worked examples

Common mistakes

Flashcards

Sigmoid function apna input kahan map karta hai?

Open interval

(0, 1)

mein, jise probability ki tarah interpret kiya jaata hai.

Sigmoid formula likhiye.

σ (z) = \frac{1}{1 + e ^{- z}}

σ (0)

kya hai?

0.5

Derivative

σ^{'} (z)

kya hai?

σ (z) (1 - σ (z))

Logistic regression kaunsi quantity ko

x

mein linear assume karta hai?

Log-odds (logit),

ln \frac{p}{1 - p} = w^{⊤} x + b

Logistic regression ke liye squared error kyun use nahi karte?

Yeh non-convex ban jaata hai aur uska gradient bahut galat hone pe saturate karta hai; cross-entropy convex hai.

Ek example ke liye binary cross-entropy loss batao.

- [y ln p + (1 - y) ln (1 - p)]

jahan

p = σ (z)

Gradient

\partial J / \partial w_{j}

kya hai?

\frac{1}{m} \sum_{i} (p_{i} - y_{i}) x_{ij}

— (pred−target)×input.

Decision boundary kahan hai?

Jahan

w^{⊤} x + b = 0

(equivalently

p = 0.5

); yeh linear hai.

Agar log-odds

= 1.1

, toh odds aur

p

kya hain?

Odds

= e^{1.1} \approx 3

, toh

p = 0.75

Logistic regression classification hai ya regression?

Classification (naam historical hai).

Recall Feynman: ek 12-saal ke bachhe ko explain karo

Socho ek machine hai jo "haan-ness score" deti hai — bada positive matlab "definitely haan", bada negative matlab "definitely naa", zero matlab "pata nahi". Lekin 7 ya −3 jaisa score confusing hai. Toh hum use ek special squishy slide (sigmoid) se pass karte hain jo koi bhi score leke 0 aur 1 ke beech ki koi number deta hai — jaise confidence ka percentage. Score 0 → 50%. Bada score → 100% ke karib. Phir hum machine ko seekhate hain uske dials nudge karke jab bhi woh bahut confident hoti hai aur galat hoti hai, is simple rule se: "kitna galat tha × input".

Connections

Linear Regression — woh linear score $z = w^{⊤} x + b$ deta hai jise sigmoid squash karta hai.
Cross-Entropy Loss — maximum likelihood se derive kiya gaya training objective.
Gradient Descent — optimizer jo $(p - y) x$ gradient use karta hai.
Maximum Likelihood Estimation — woh principle jo loss ko justify karta hai.
Softmax Regression — sigmoid ka multi-class generalization.
Decision Boundaries — kyun boundary yahan ek hyperplane hai.
Neural Networks — sigmoid as an activation function / ek single logistic neuron.