Logistic regression — sigmoid, cross-entropy loss: Visual walkthrough

5.6.2 · D2 · Coding › Machine Learning (Aerospace Applications) › Logistic regression — sigmoid, cross-entropy loss

Yeh page logistic regression ke poore engine ko pictures ki ek sequence ke roop mein rebuild karti hai. Hum ek single haan/naa question se shuru karte hain aur us ek gradient formula tak pahunchte hain jo sab kuch train karta hai. Har symbol ko use karne se pehle draw kiya jaata hai.

Step 1 — Ek aisa sawaal jiska sirf do jawaab hain

KYA HAI. Ek turbine blade imagine karo. Hum ek hi sawaal poochte hain: kya yeh fail hogi? Sirf do jawaab hain: haan (hum ise number $y = 1$ likhte hain) aur naa (hum $y = 0$ likhte hain).

KYUN. Baad ki har cheez ko aim karne ke liye ek target chahiye. Numbers $1$ aur $0$ use karna (words "haan"/"naa" ki jagah) hume baad mein jawaab par arithmetic karne deta hai — ise subtract karo, multiply karo, average karo.

PICTURE. Ek vertical axis par do dots: $1$ height par ek red dot ("fail"), aur $0$ height par ek black dot ("safe"). Beech mein kuch nahi — ek real blade ya toh fail hui ya nahi.

Step 2 — Raw score $z$ poori number line par rehta hai

KYA HAI. Haara model ek feature $x$ dekhta hai (maano, operating hours) aur ek plain number compute karta hai: $z = how much x matters w \cdot the measurement x + baseline shift b$

KYUN. Yeh hi ek cheez hai jo machine ke liye sasti hai: multiply aur add. Lekin disaster dekho — $z$ kuch bhi ho sakta hai: $- 40$ , $0$ , $+ 7000$ . Yeh probability nahi hai. Ek probability $0$ aur $1$ ke beech honi chahiye. Humein ek mismatch fix karni hai.

PICTURE. Ek horizontal number line jo $- \infty$ se $+ \infty$ tak stretch karti hai, jisme $z$ freely slide karta hai. Red segment sirf woh region mark karti hai jahan probability ko rehne ki ijazat hai — $0$ se $1$ tak ka ek tiny sliver — jo dikhata hai ki $z$ kitna zyada overshoot karta hai.

Step 3 — Odds kyun natural bridge hai

KYA HAI. Hum free number $z$ ko probability $p$ (jawaab ke $1$ hone ki chance) se odds use karke connect karte hain: $odds = \frac{p}{1 - p}$ aur hum assume karte hain ki model odds ka log output karta hai: $lo g (\frac{p}{1 - p}) = z .$

KYUN. Odds kyun, $p$ khud kyun nahi? Ek probability $[0, 1]$ mein trapped hai, lekin odds $0$ (pakka "naa") se $+ \infty$ (pakka "haan") tak jaate hain. Logarithm lo aur range dono taraf explode ho jaati hai: $- \infty$ se $+ \infty$ — exactly woh range jisme $z$ pehle se rehta hai. Toh log-odds woh ek transformation hai jo "unbounded number" aur "probability" ko ek hi bhasha mein baat karne deta hai. $lo g$ choose karna (koi aur stretch nahi) hi woh cheez hai jo feature effects ko linearly add up karti hai.

PICTURE. Teeno stacked bars jo ek hi belief ko teen languages mein dikhate hain: probability $p = 0.75$ (ek bar jo 1 par cap hai), odds $= 3$ (ek lamba bar), log-odds $\approx 1.1$ (ek two-way axis par signed value). Red arrows har transform ko range wider stretch karte dikhate hain.

Step 4 — $p$ ke liye solve karo: sigmoid force hokar exist karta hai

KYA HAI. Ab hum log-odds undo karte hain $p$ wapas paane ke liye. $\frac{p}{1 - p} = e^{z}$ se shuru karte hain (dono sides exponentiate karo — $e^{(\cdot)}$ $lo g$ ko cancel karta hai): $p = e^{z} (1 - p) \Rightarrow p + p e^{z} = e^{z} \Rightarrow p (1 + e^{z}) = e^{z} \Rightarrow p = \frac{e ^{z}}{1 + e ^{z}} .$ Upar aur neeche $e^{- z}$ se multiply karo: $p = σ (z) = \frac{1}{1 + e ^{- z}}$

KYUN. Humne yeh S-curve fun ke liye invent nahi ki — yeh wahi ek function hai jo "log-odds linear hai" ke saath consistent hai. Term by term: $e^{- z}$ $0$ tak shrink ho jaata hai jab $z \to + \infty$ (toh $σ \to 1$ ), aur blow up ho jaata hai jab $z \to - \infty$ (toh $σ \to 0$ ). Denominator mein $+ 1$ guarantee karta hai ki hum kabhi zero se divide nahi karte aur $1$ se kabhi exceed nahi karte.

PICTURE. Famous S: ek smooth red curve height $0$ se (far left) height $1$ tak (far right), bilkul center mein $(0, 0.5)$ se guzarti hui. Flat tails aur steep middle label kiye hue hain.

Saare cases covered.

$z = 0 \Rightarrow σ = \frac{1}{1 + 1} = 0.5$ — fence-sitting decision boundary.
$z \to + \infty \Rightarrow e^{- z} \to 0 \Rightarrow σ \to 1$ — "haan" mein total confidence.
$z \to - \infty \Rightarrow e^{- z} \to \infty \Rightarrow σ \to 0$ — "naa" mein total confidence.

Step 5 — S ka slope ek chupi hui gift hai

KYA HAI. S kitni tez chadhti hai? Differentiate karo (chain rule on $u = 1 + e^{- z}$ ): $σ^{'} (z) = \frac{e ^{- z}}{( 1 + e ^{- z} ) ^{2}} = σ (z) \frac{1}{1 + e ^{- z}} \cdot 1 - σ (z) \frac{e ^{- z}}{1 + e ^{- z}} = σ (z) (1 - σ (z)) .$

KYUN. Humein slope chahiye kyunki gradient descent downhill move karta hai, aur "downhill" matlab "slope ki direction mein". Magic yeh hai ki slope purely output $σ$ ke terms mein express hota hai — koi leftover exponentials nahi. Isliye training sasti hai: ek baar jab aap probability jaante ho, aap turant slope jaante ho.

PICTURE. S-curve phir se black mein, uske saath slope curve $σ (1 - σ)$ neeche red mein draw ki — ek bell-shaped hump jo $z = 0$ par peak karti hai (value $0.25$ ) aur dono tails par gayab ho jaati hai, jo dikhata hai kahaan curve steep hai vs. flat.

Step 6 — Surprise score karna: cross-entropy

KYA HAI. Humein ab ek number chahiye jo kahe "guess $\overset{y}{^} = σ (z)$ truth $y$ ke against kitni buri thi?" Score hai: $L (y, \overset{y}{^}) = - [y lo g \overset{y}{^} + (1 - y) lo g (1 - \overset{y}{^})] .$

KYUN. Dekho kaise dono labels ise on aur off karte hain. Term by term:

Agar $y = 1$ : $(1 - y)$ term vanish ho jaata hai, $L = - lo g \overset{y}{^}$ bachta hai.
Agar $y = 0$ : $y$ term vanish ho jaata hai, $L = - lo g (1 - \overset{y}{^})$ bachta hai.

Sirf woh term survive karti hai jo sach jawaab se match karti hai. Aur $- lo g (truth ki probability)$ exactly "surprise" hai: agar tumne truth ko probability $1$ di, surprise $- lo g 1 = 0$ hai; agar tumne use probability $0$ ke paas di, surprise $+ \infty$ tak rocket karta hai. Confident wrong answers ko mercilessly punish kiya jaata hai.

PICTURE. Do curves: red hai $- lo g (\overset{y}{^})$ (cost jab $y = 1$ ) jo $0$ ki taraf dive karti hai jab $\overset{y}{^} \to 1$ aur upar shoot karti hai jab $\overset{y}{^} \to 0$ ; black hai $- lo g (1 - \overset{y}{^})$ (cost jab $y = 0$ ), iska mirror image.

Step 7 — Exactly yeh score kyun? Maximum likelihood

KYA HAI. Assume karo har label ek coin se aayi jo probability $\overset{y}{^}_{i}$ se biased hai (ek Bernoulli trial): $P (y_{i} ∣ \overset{y}{^}_{i}) = \overset{y}{^}_{i}^{y_{i}} (1 - \overset{y}{^}_{i})^{1 - y_{i}} .$ Saara data dekhne ki chance (independent) product hai; uska log lo aur negate karo: $J = - \frac{1}{N} \sum_{i = 1}^{N} [y_{i} lo g \overset{y}{^}_{i} + (1 - y_{i}) lo g (1 - \overset{y}{^}_{i})] .$

KYUN. Humne cross-entropy taste se nahi choose ki — yeh hai negative log-likelihood. Likelihood maximise karna = "woh weights choose karo jo data jo humne actually dekha use least surprising banate hain". $lo g$ ek unwieldy product ko ek friendly sum mein badal deta hai (sums term-by-term differentiate hote hain); minus "maximise" ko "minimise" mein badal deta hai, toh hum ise gradient descent ko de sakte hain. Yeh maximum-likelihood view hai.

PICTURE. Bernoulli coins ki ek row, har ek ka apna bias $\overset{y}{^}_{i}$ , ek lambi product (red) mein multiply hote hue, phir ek downward arrow ussi cheez ki taraf jo sum of logs ke roop mein rewrite hai — visually ek scary product ko ek tidy sum mein shrink karte hue.

Step 8 — Miracle cancellation: gradient

KYA HAI. Humein loss ka slope chahiye weight $w_{j}$ ke respect mein. Teen pieces chain karo: $\frac{\partial L}{\partial w _{j}} = - \frac{y}{y ^} + \frac{1 - y}{1 - y ^} \frac{\partial L}{\partial y ^} \cdot \overset{y}{^} (1 - \overset{y}{^}) \frac{\partial y ^}{\partial z} \cdot x_{j} \frac{\partial z}{\partial w _{j}} .$ Pehle do multiply karo: ugly fractions Step 5 ke $\overset{y}{^} (1 - \overset{y}{^})$ se cancel ho jaate hain: $[- y (1 - \overset{y}{^}) + (1 - y) \overset{y}{^}] = - y + y \overset{y}{^} + \overset{y}{^} - y \overset{y}{^} = \overset{y}{^} - y .$ Toh: $\frac{\partial L}{\partial w _{j}} = (\overset{y}{^} - y) x_{j}$

KYUN. Yahi wajah hai ki humne sigmoid aur cross-entropy dono ek saath chose ki: sigmoid ka slope $\overset{y}{^} (1 - \overset{y}{^})$ (jo Step 5 mein saturation cause karta tha) exactly woh cheez hai jo $\partial L / \partial \overset{y}{^}$ ke numerator mein appear hoti hai — aur woh cancel ho jaati hai. Freeze theek ho gayi. Jo bacha woh pure meaning hai: error $(\overset{y}{^} - y)$ times feature $x_{j}$ . Galat-aur-confident? Bada error, bada correction.

PICTURE. Teen-box chain diagram: box $\partial L / \partial \overset{y}{^}$ , box $\overset{y}{^} (1 - \overset{y}{^})$ , box $x_{j}$ ek arrow feed karte hue; do middle terms red mein crossing out dikhaye hue, exit par $(\overset{y}{^} - y) x_{j}$ bachta hai.

Degenerate cases.

Perfect prediction $\overset{y}{^} = y$ : gradient $= 0$ — kuch fix nahi, learning sahi tarah se ruk jaati hai.
$\overset{y}{^} \to 0$ ya $1$ exactly: $lo g$ numerically blow up hota hai → code mein $\overset{y}{^}$ ko $[ε, 1 - ε]$ mein clamp karo (aur Regularization (L1, L2) dekho weights ko $\pm \infty$ tak race karne se rokne ke liye).

Ek-picture summary

KYA HAI. Ek figure jo saatein steps ko thread karta hai: feature $x$ → linear score $z$ → sigmoid $\overset{y}{^}$ mein squash karta hai → truth $y$ se compare karo → cross-entropy surprise → slope $(\overset{y}{^} - y) x$ mein collapse hota hai → $w$ ko nudge karo.

Recall Feynman retelling — kisi dost ko samjhao jaise

Ek machine sirf add aur multiply kar sakti hai, toh pehle woh ek plain number $z = w x + b$ banati hai. Woh number kuch bhi ho sakta hai, lekin humein ek probability chahiye thi, jo 0 aur 1 ke beech honi chahiye. Trick yeh hai: assume karo ki $z$ odds ka log hai, kyunki log-odds woh ek cheez hai jo poori number line par range karti hai bilkul $z$ ki tarah. Us assumption ko algebraically undo karo aur S-shaped sigmoid nikal aata hai — humne ise invent nahi kiya, yeh force hua. Sigmoid ka slope $\overset{y}{^} (1 - \overset{y}{^})$ nikalta hai, sirf apne output ke terms mein likha hua. Phir humein ek report card chahiye: cross-entropy, jo basically hai "mujhe sach wala label dekhke kitni surprise hui?" aur yeh coin-flip model ki negative log-likelihood hoti hai, toh mathematically honest choice hai. Aakhir mein, jab hum poochte hain ki report card kaise badlti hai jab hum ek weight tweak karte hain, toh sigmoid ka slope messy fractions ko cancel kar deta hai, aur sab kuch melt ho jaata hai error times feature, $(\overset{y}{^} - y) x$ mein. Wahi simplicity poori wajah hai ki logistic regression fast aur stable hai.

Related destinations: Softmax Regression (kai classes), Neural Networks (yeh units stack karo), ROC Curves and AUC (classifier ko judge karo), Bayesian Interpretation ( $w$ par priors).