Foundations — Logistic regression — sigmoid, cross-entropy loss

5.6.2 · D1 · Coding › Machine Learning (Aerospace Applications) › Logistic regression — sigmoid, cross-entropy loss

Pehle aap parent topic ki ek bhi line padh sakein, aapko pehle se ek dozen chhote tools apne paas rakhne chahiye. Yeh page unhe ek-ek karke bilkul scratch se banata hai. Upar se neeche padho — har tool agla tool istemal karta hai.

0. "Classification" ka matlab kya hota hai

Picture yeh sochein: parts ko ek conveyor belt par do bins mein sort karna. Koi "1.5 bin" nahi hota. Ise altitude predict karne se compare karein (jo ek continuous number hai) — woh regression hoga, Linear Regression ka kaam. Logistic regression uska classification cousin hai.

Topic ko yeh kyun chahiye: poori machine ek aisa number output karne ke liye bani hai jise hum ek label se compare kar sakein jo sirf 0 ya 1 hoti hai.

1. Numbers, variables, aur subscripts

Chaliye parent page ke alphabet ko fix karte hain, kyunki woh chhote letters mein asli meaning chhupata hai.

Picture: apne dataset ko ek spreadsheet ki tarah sochein. Rows examples hain (superscript $i$). Columns features hain (subscript $j$). Ek cell hai $x_j^{(i)}$.

2. Vectors aur dot product

Parent seedha ${\mathbf{w}}^T\mathbf{x}$ par jump karta hai. Chaliye ise earn karte hain.

Neeche ki picture ek 2-feature example ko ek plane mein arrow ki tarah dikhati hai, aur weight vector $\mathbf{w}$ ko bhi ussi space mein dikhati hai.

Yeh tool kyun aur koi nahi? Hume ek single number chahiye jo saare features ko ek "confidence score" mein summarize kare. Dot product sabse simple tarika hai kai inputs ko ek mein blend karne ka, har input ke liye ek tunable knob ($w_j$) ke saath. Us single number ka apna ek naam hota hai:

3. Exponential $e^{-z}$ — bending tool

Topic ko specifically $e^{-z}$ kyun chahiye? Hume ek aisa knob chahiye jo:

• kabhi negative na ho (probabilities negative denominators se nahi aa sakti),
• smoothly "bahut chhota" se "bahut bada" connect kare jab $z$ number line par sweep kare.

$e^{-z}$ exactly yahi karta hai: jab $z$ bada hota hai, $e^{-z}\to 0$; jab $z$ bahut negative hota hai, $e^{-z}\to \infty$. Amber curve dekho — wahi behaviour hai jis par sigmoid sawa hai.

4. Sab milaate hain: sigmoid shape

Ab parent ka headline formula $\sigma (z)=\frac{1}{1+e^{-z}}$ readable hai. Aap har piece pehle se jaante hain: $z$ raw score hai, $e^{-z}$ hamesha-positive bending tool hai, aur 1 ko "1 + kuch positive" se divide karna answer ko $(0,1)$ mein force karta hai.

• Jab $z$ bahut bada positive ho: $e^{-z}\approx 0$, toh $\sigma \approx \frac{1}{1}=1$.
• Jab $z=0$: $e^0=1$, toh $\sigma =\frac{1}{2}=0.5$ — decision boundary (figure mein cyan cross).
• Jab $z$ bahut negative ho: $e^{-z}$ bahut bada, toh $\sigma \approx 0$.

Hum smooth S-shape ki parwah kyun karte hain na ki 0 par hard step ki: Gradient Descent ko weights ko kis taraf nudge karna hai yeh jaanne ke liye har jagah slope chahiye. Vertical cliff ka koi usable slope nahi hota. Gentle S ka har point par slope hota hai.

5. Odds, probability, aur logarithm

Parent sigmoid ko "odds" se derive karta hai. Do aur tools.

Picture: probability 0 se 1 tak ka slider hai; odds usi slider ko stretch karke 0 se $+\infty$ tak le jaata hai.

Wahi last property exactly wajah hai ki parent likelihood (ek bada product) ka $\log$ leta hai — woh ek friendly sum ban jaata hai. "Log-odds" $\log \frac{p}{1-p}$ probability ko poori number line par stretch karta hai, isliye woh unbounded score $z$ ke barabar ho sakta hai.

6. Derivatives — "downhill kaun si taraf hai" tool

Picture: ek hillside par khade ho (loss surface). Derivative woh arrow hai jo seedha aapke pair ke neeche downhill point karta hai. Gradient Descent bas baar baar us arrow ki opposite direction mein step karta hai valley floor (minimum loss) tak pohunchne ke liye.

Topic ko derivatives kyun chahiye: slope ke bina koi tarika nahi hai yeh jaanne ka ki $\mathbf{w}$ aur $b$ ko kaise change karein taaki loss chhota ho. Training slopes follow karna hi hai.

7. Cost/loss symbols $L$ aur $J$

Picture: $J$ woh landscape ki height hai jise Gradient Descent neeche utarta hai; har training step ise chhota karta hai. Yeh Maximum Likelihood Estimation se connect hota hai — cross-entropy minimize karna exactly waisi weights chunne jaisa hai jo observed labels ko sabse zyada likely banaati hain.

8. Learning rate $\alpha$

Picture: $\alpha$ aapki stride length hai downhill chalte waqt. Bahut bada ho toh aap valley ke paar kood jaate ho aur bounce karte ho; bahut chhota ho toh forever crawl karte ho. Parent ek chhota $\alpha =10^{-6}$ isliye choose karta hai kyunki unnormalized features slope bahut bada bana dete hain.

Sab kuch topic mein kaise fit hota hai

Ise ek loop ki tarah padho: features aur weights $z$ banate hain; sigmoid $z$ ko $\hat{y}$ mein badalta hai; $\hat{y}$ ko $y$ se compare karne par loss $J$ milta hai; derivatives downhill direction dete hain; gradient descent weights update karta hai; repeat.

Equipment checklist

Right side cover karo aur zor se jawab do. Agar koi bhi atak jaaye, uska section upar phir se padho.