Worked examples — Logistic regression — sigmoid, cross-entropy loss

5.6.2 · D3 · Coding › Machine Learning (Aerospace Applications) › Logistic regression — sigmoid, cross-entropy loss

Yeh Logistic Regression ka ek worked-example deep dive hai. Parent note ne machinery banayi thi — sigmoid, cross-entropy, gradient $(\hat{y}-y)x_j$. Yahan hum ise har us case ke against stress-test karenge jo yeh topic throw kar sakta hai: $z$ ki har sign, degenerate $\hat{y}=0.5$ boundary, saturated tails jahan gradients vanish ho jaate hain, ek real aerospace word problem, aur ek exam-style twist.

Kuch naya assume nahi kiya gaya hai. Agar koi symbol aata hai, toh woh parent mein earn kiya ja chuka hai. Pehle cases ko naam lete hain.

---

The scenario matrix

Har logistic-regression computation inn cells mein se kisi ek mein rehti hai. Hum sab ko cover karenge.

Cell	Kya ise alag banata hai	Covered by
A. $z>0$, correct	model confident aur sahi	Ex 1
B. $z<0$, wrong	model confident aur galat — bada loss	Ex 2
C. $z=0$ boundary	$\hat{y}=0.5$ exactly — decision knife-edge	Ex 3
D. Saturated tail	$|z|$ bahut bada — sigmoid flat, gradient $\to 0$	Ex 4
E. Multi-feature vector	pura ${\mathbf{w}}^{\top }\mathbf{x}+b$ mixed signs ke saath	Ex 5
F. Full descent step	forward → loss → gradient → weight update	Ex 6
G. Real-world word problem	English → features → prediction translate karna	Ex 7
H. Exam twist	odds/log-odds reasoning, koi calculator sigmoid nahi	Ex 8

Upar S-curve dekho. Chaar coloured dots is poore page ke characters hain: green (confident-right, cell A), red (confident-wrong, cell B), yellow (the boundary, cell C), aur faded dots jo tails par bahut door hain (saturation, cell D). Yeh picture apne dimaag mein rakho.

---

Example 1 — Cell A: $z>0$, prediction correct

Forecast: compute karne se pehle guess karo — $z$ positive lagta hai, $y=1$, toh hum expect karte hain $\hat{y}>0.5$ aur chhota loss. Apna guess likh lo.

1. Logit $z$ compute karo. $z={\mathbf{w}}^{\top }\mathbf{x}+b=0.5(2)+0.5(3)-1=1+1.5-1=1.5$. Yeh step kyun? Aage ka sab kuch $z$ chahiye; yeh ek akela number hai jo saari feature information sigmoid mein le jaata hai.
2. Squash karo. $\hat{y}=\sigma (1.5)=\frac{1}{1+e^{-1.5}}=\frac{1}{1+0.2231}=0.8176$. Yeh step kyun? Sigmoid raw score $z=1.5$ ko probability mein badalta hai. $z>0\Rightarrow \hat{y}>0.5$, bilkul forecast ke anusaar.
3. Loss. Kyunki $y=1$, sirf pehla term bachta hai: $L=-\log (0.8176)=0.2014$. Yeh step kyun? $y=1$ ke saath cross-entropy $-\log \hat{y}$ hai; 1 ke paas probability se loss 0 ke paas aata hai.

Verify karo: $\hat{y}=0.8176>0.5$ ✓ (class 1 predict karta hai, truth se match). Loss $0.20$ chhota hai ✓. Sanity: $-\log (0.82)\approx 0.20$, aur 1 se thoda kam number ka $\log$ ek chhota negative hai, toh $-\log$ chhota positive hai. Forecast confirm hua.

---

Example 2 — Cell B: $z<0$, prediction WRONG

Forecast: truth cloudy hai ($y=1$) lekin agar $z$ negative nikle toh model "not cloudy" vote karta hai — ek confident galat jawab, toh expect karo bada loss.

1. Logit. $z=0.01(180)+(-0.02)(45)-1.5=1.8-0.9-1.5=-0.6$. Yeh step kyun? Mixed-sign weights: bright pixels upar push karte hain, variance term neeche. Net $z=-0.6$ negative hai.
2. Squash. $\hat{y}=\sigma (-0.6)=\frac{1}{1+e^{0.6}}=\frac{1}{1+1.8221}=0.3543$. Yeh step kyun? $z<0\Rightarrow \hat{y}<0.5$: model kehta hai "probably not cloudy."
3. Loss. $y=1$, toh $L=-\log (0.3543)=1.0376$. Yeh step kyun? Cross-entropy confident wrongness ko punish karta hai — $0.35$ par hona jab jawab 1 hai, Example 1 ke loss ka lagbhag 5× cost karta hai.

Verify karo: $\hat{y}<0.5$ lekin $y=1$ → misclassified ✓. Losses compare karo: $1.04$ (galat) vs $0.20$ (sahi) — galat case bahut zyada hurt karta hai, exactly wahi jo ek achha loss karna chahiye ✓.

---

Example 3 — Cell C: the $z=0$ boundary (degenerate)

Forecast: $z$ exactly $0$ hoga, toh $\hat{y}=0.5$ — model maximally uncertain hai. Gradient kaisa dikhta hai jab model commit karne se mana kar de?

1. Logit. $z=2(1)-2=0$. Yeh step kyun? Yeh decision boundary hai: $\sigma (0)=0.5$ woh jagah hai jahan "class 0" aur "class 1" tie karte hain.
2. Squash. $\hat{y}=\sigma (0)=\frac{1}{1+e^0}=\frac{1}{2}=0.5$. Yeh step kyun? $e^0=1$, toh sigmoid exactly aadha deta hai — upar figure mein yellow dot.
3. Loss. $y=0$, toh $L=-\log (1-\hat{y})=-\log (0.5)=0.6931$. Yeh step kyun? Boundary par loss $\log 2\approx 0.693$ hai chahe true label kuch bhi ho — total uncertainty dono taraf ek jaisi cost karti hai.
4. Gradient. $\partial L/\partial w=(\hat{y}-y)x=(0.5-0)(1)=0.5$. Yeh step kyun? Positive gradient → descent $w$ ko decrease karega, $z$ ko 0 se neeche push karega taaki $\hat{y}$ correct class 0 ki taraf drop ho.

Verify karo: $\hat{y}=0.5$ exactly ✓. Loss $=\log 2=0.693$ ✓. Gradient boundary par non-zero hai — isliye logistic regression ambiguous points se bhi seekhta rehta hai, ek hard threshold ke unlike.

---

Example 4 — Cell D: saturation, jahan gradients vanish ho jaate hain

Forecast: $z=8$ bahut bada hai, toh $\hat{y}$ $\approx 1$ par chipka hua hai. Jab model sahi ho, expect karo tiny gradient. Jab yeh $z=8$ par galat ho (ek confident blunder), gradient utna hi bada ho sakta hai jitna feature allow karta hai — dekho.

1. Logit aur squash. $z=8$, $\hat{y}=\sigma (8)=\frac{1}{1+e^{-8}}=\frac{1}{1+0.000335}=0.99966$. Yeh step kyun? Sigmoid yahan essentially flat hai (figure mein right side par near-horizontal curve dekho).
2. Gradient, correct label $y=1$. $\partial L/\partial w=(\hat{y}-1)x=(0.99966-1)(1)=-0.000335$. Yeh step kyun? Almost zero. Model sahi aur confident hai, toh seekhne ke liye kuch nahi — descent barely move karta hai. Yeh vanishing-gradient tail hai.
3. Gradient, wrong label $y=0$. $\partial L/\partial w=(\hat{y}-0)x=0.99966(1)=0.99966$. Yeh step kyun? Ek confident galat prediction maximum possible gradient magnitude deti hai (feature ka near 1×) — model ko zor se wapas khaincha jaata hai.

Verify karo: $\sigma (8)\approx 0.9997$ ✓. Correct-label gradient $\approx -3.35\times 10^{-4}$ (tiny) ✓; wrong-label gradient $\approx 1.0$ (large) ✓. Gradient magnitude $|\hat{y}-y|$ 1 se bounded hai — saturation learning speed ko limit karta hai jab sahi ho, lekin kabhi nahi jab catastrophically galat ho.

---

Example 5 — Cell E: multi-feature vector, mixed signs

Forecast: teen features alag-alag directions mein khichti hain; bias add karo. Numbers crunch karne se pehle $z$ ka sign guess karo.

1. Dot product term by term. ${\mathbf{w}}^{\top }\mathbf{x}=(1.0)(1.2)+(2.0)(-0.5)+(-0.5)(3.0)=1.2-1.0-1.5=-1.3$. Yeh step kyun? Dot product ek weighted vote hai: har feature apne weight se multiply hoti hai, phir sum hota hai. Negative weights ek feature ko class 0 ke liye vote karne dete hain.
2. Bias add karo. $z=-1.3+0.4=-0.9$. Yeh step kyun? Bias $b$ poori decision boundary shift karta hai; yahan yeh negative vote ko thoda soft karta hai.
3. Squash. $\hat{y}=\sigma (-0.9)=\frac{1}{1+e^{0.9}}=\frac{1}{1+2.4596}=0.2891$.
4. Loss. $y=0$, toh $L=-\log (1-\hat{y})=-\log (0.7109)=0.3412$. Yeh step kyun? $y=0$ matlab hum chahte hain $\hat{y}$ chhota ho; $0.29$ reasonably chhota hai, toh loss modest hai.

Verify karo: $z=-0.9<0\Rightarrow \hat{y}<0.5$ ✓, aur truth class 0 hai, toh model sahi hai → chhota loss $0.34$ ✓.

---

Example 6 — Cell F: ek complete gradient-descent step

Forecast: $z=0.6$ positive hai lekin confident nahi; $y=1$, toh expect karo ki step ke baad $w$ increase hoga.

1. Forward. $z=0.2(3)+0=0.6$; $\hat{y}=\sigma (0.6)=\frac{1}{1+e^{-0.6}}=0.6457$. Yeh step kyun? Current prediction establish karta hai taaki hum measure kar sakein kitna galat hai.
2. Loss. $L=-\log (0.6457)=0.4375$. Yeh step kyun? "Before" cost record karta hai taaki baad mein confirm kar sakein ki update ne help ki.
3. Gradient. $\partial L/\partial w=(\hat{y}-y)x=(0.6457-1)(3)=-1.0629$. Yeh step kyun? Negative → prediction $y=1$ ke liye bahut kam; descent ko $w$ mein add karna hai.
4. Update. $w:=w-\alpha \partial L/\partial w=0.2-0.5(-1.0629)=0.2+0.5314=0.7314$. Yeh step kyun? Gradient descent gradient ke against move karta hai; double negative $w$ ko raise karta hai.

Verify karo: New $z=0.7314(3)=2.194$, new $\hat{y}=\sigma (2.194)=0.8997$ — $0.6457$ se upar, target $1$ ke paas ✓. New loss $-\log (0.8997)=0.1057<0.4375$ ✓. Step ne loss reduce kiya, toh yeh sahi direction mein gaya.

---

Example 7 — Cell G: aerospace word problem

Forecast: dono features high hain positive weights ke saath → high risk expected. Guess: flagged.

1. English ko $z$ mein translate karo. $z=1.2(1.5)+0.9(0.8)-1.0=1.8+0.72-1.0=1.52$. Yeh step kyun? Har standardized feature apne weight se multiply hokar risk mein contribute karta hai; bias baseline threshold set karta hai.
2. Probability. $\hat{y}=\sigma (1.52)=\frac{1}{1+e^{-1.52}}=\frac{1}{1+0.2187}=0.8206$. Yeh step kyun? Risk score ko ek interpretable failure probability mein convert karta hai — $82\%$.
3. Decision. $\hat{y}=0.8206\ge 0.5$ → inspection ke liye flag karo. Yeh step kyun? Operational threshold apply karta hai; yahan safety jeetti hai.

Verify karo: $z>0\Rightarrow \hat{y}>0.5$ ✓, decision "high hours + high vibration = risky" ke saath consistent ✓. Failure probability $\approx 82\%$ — $z=1.52$ plug karo: $e^{-1.52}\approx 0.219$, $1/1.219\approx 0.820$ ✓.

---

Example 8 — Cell H: exam twist (odds & log-odds, koi sigmoid table nahi)

Forecast: parent note ne dikhaya tha $\log (\text{odds})=z$. Toh odds $=e^z$, aur $z$ mein add karna odds ko multiply karta hai. Guess: doubling-ish.

1. Log-odds se odds. $\text{odds}=e^z=e^{0.7}=2.0138$. Yeh step kyun? By definition $\log (p/(1-p))=z$, toh exponentiate karne se log undo ho jaata hai. Odds $\approx 2:1$ cloudy ke favour mein.
2. Odds se probability. $\hat{y}=\frac{\text{odds}}{1+\text{odds}}=\frac{2.0138}{3.0138}=0.6682$. Yeh step kyun? $p/(1-p)=\text{odds}$ rearrange karne par $p=\text{odds}/(1+\text{odds})$ milta hai — yeh $\sigma (0.7)$ ke equal hai, lekin humein koi sigmoid table nahi chahiye tha.
3. $+0.7$ ka odds par effect. New odds $=e^{z+0.7}=e^z\cdot e^{0.7}=\text{old odds}\times 2.0138$. Yeh step kyun? Yahi log-odds ke additive hone ka poora point hai: $z$ mein ek fixed bump odds ko ek constant factor $e^{0.7}\approx 2.01$ se multiply karta hai, chahe aap kahin se bhi shuru karo.

Verify karo: $\sigma (0.7)=1/(1+e^{-0.7})=1/(1+0.4966)=0.6682$ ✓ — step 2 se exactly match. Odds-multiplier $e^{0.7}=2.0138$ ✓. Yeh odds interpretation hai jo coefficients ko readable banati hai: weight $w_j$ matlab "har unit $x_j$ odds ko $e^{w_j}$ se multiply karta hai."

---

Recall Sab cells mein quick self-test

Kaun se cell mein $\hat{y}=0.5$ exactly hota hai? ::: Cell C, boundary $z=0$. Gradient vanishingly small kab hota hai? ::: Cell D — saturated tail AUR already correct. $z$ mein $\Delta$ add karne se odds ka kya hota hai? ::: Unhe $e^{\Delta }$ se multiply karta hai (Cell H). Ek confident galat prediction phir bhi fast kyun seekhti hai? ::: Uski gradient magnitude $|\hat{y}-y|$ 1 ke paas hai, maximum. One-line gradient formula? ::: $\partial J/\partial w_j=(\hat{y}-y)x_j$ examples par average kiya hua.