Logistic regression — sigmoid, cross-entropy loss

5.6.2 · Coding › Machine Learning (Aerospace Applications)

Overview

Logistic regression ek fundamental algorithm hai binary classification problems ke liye aerospace ML mein. Yeh continuous values predict karne ke bajaye (jaise altitude), probabilities predict karta hai (jaise "kya yeh component fail hoga?" ya "kya yeh satellite image cloudy hai?"). Yeh kisi bhi real-valued input ko 0 aur 1 ke beech ek probability mein map karta hai sigmoid function use karke, phir cross-entropy loss minimize karke train hota hai.

Figure — Logistic regression — sigmoid, cross-entropy loss

Core Components

Sigmoid function exactly yahi karta hai. Ise ek "probability translator" samjho — yeh model ka raw confidence score leta hai aur use ek valid probability mein convert karta hai.

1. The Sigmoid Function

First Principles se Derivation:

Yeh specific form kyun? Odds ratio se shuru karo. Agar class 1 ki probability $p$ hai, toh odds hain: $odds = \frac{p}{1 - p}$

Logistic regression mein, hum maante hain ki log-odds features mein linear hai: $lo g (\frac{p}{1 - p}) = z = w^{T} x + b$

Kyun? Yeh key assumption hai — features ka effect odds par multiplicative hota hai, lekin log-odds par additive. Ab $p$ ke liye solve karo:

$\frac{p}{1 - p} = e^{z}$ $p = e^{z} (1 - p)$ $p = e^{z} - p e^{z}$ $p (1 + e^{z}) = e^{z}$ $p = \frac{e ^{z}}{1 + e ^{z}}$

Numerator aur denominator dono ko $e^{- z}$ se multiply karo: $p = \frac{1}{e ^{- z} + 1} = σ (z)$

Key Properties:

$σ (0) = 0.5$ (decision boundary)
$σ (z) \to 1$ jab $z \to \infty$
$σ (z) \to 0$ jab $z \to - \infty$
Derivative: $σ^{'} (z) = σ (z) (1 - σ (z))$ (backprop ke liye bahut convenient!)

Derivative ki Derivation: $\frac{d}{d z} σ (z) = \frac{d}{d z} (\frac{1}{1 + e ^{- z}})$ $u = 1 + e^{- z}$ ke saath chain rule use karke: $= - \frac{1}{u ^{2}} \cdot (- e^{- z}) = \frac{e ^{- z}}{( 1 + e ^{- z} ) ^{2}}$ $= \frac{1}{1 + e ^{- z}} \cdot \frac{e ^{- z}}{1 + e ^{- z}} = σ (z) \cdot \frac{e ^{- z}}{1 + e ^{- z}}$ $= σ (z) \cdot (1 - \frac{1}{1 + e ^{- z}}) = σ (z) (1 - σ (z))$

Yeh step kyun? Derivative ka yeh form gradient calculations ko elegant banata hai — gradient updates mein koi exponentials nahi!

2. Cross-Entropy Loss

Information theory ka cross-entropy exactly yahi karta hai.

$N$ training examples ke liye: $J (w, b) = - \frac{1}{N} \sum_{i = 1}^{N} [y^{(i)} lo g (\overset{y}{^}^{(i)}) + (1 - y^{(i)}) lo g (1 - \overset{y}{^}^{(i)})]$

Maximum Likelihood se Derivation:

Maano ki har label $y_{i}$ , probability $\overset{y}{^}_{i}$ wali Bernoulli distribution se aata hai: $P (y_{i} ∣ \overset{y}{^}_{i}) = \overset{y}{^}_{i}^{y_{i}} (1 - \overset{y}{^}_{i})^{1 - y_{i}}$

Yeh form kyun? Jab $y_{i} = 1$ , yeh $\overset{y}{^}_{i}$ hai. Jab $y_{i} = 0$ , yeh $1 - \overset{y}{^}_{i}$ hai. Ek perfect compact notation.

Saare data ki likelihood (independence maanke): $L = \prod_{i = 1}^{N} \overset{y}{^}_{i}^{y_{i}} (1 - \overset{y}{^}_{i})^{1 - y_{i}}$

Log-likelihood lo (products sums ban jaate hain, optimize karna aasaan): $lo g L = \sum_{i = 1}^{N} [y_{i} lo g \overset{y}{^}_{i} + (1 - y_{i}) lo g (1 - \overset{y}{^}_{i})]$

Log-likelihood maximize karo = negative log-likelihood minimize karo: $J = - \frac{1}{N} lo g L = - \frac{1}{N} \sum_{i = 1}^{N} [y_{i} lo g \overset{y}{^}_{i} + (1 - y_{i}) lo g (1 - \overset{y}{^}_{i})]$

Yeh step kyun? Likelihood maximize karna woh parameters dhundta hai jo observed data ke saath sabse consistent hain. Negative sign ise ek aisi loss banata hai jise minimize kiya ja sake.

Intuition Check:

Agar $y = 1$ aur $\overset{y}{^} = 0.9$ : loss = $- lo g (0.9) = 0.105$ (chhota)
Agar $y = 1$ aur $\overset{y}{^} = 0.1$ : loss = $- lo g (0.1) = 2.303$ (bada!)
Agar $y = 0$ aur $\overset{y}{^} = 0.9$ : loss = $- lo g (0.1) = 2.303$ (bada!)

$\frac{\partial J}{\partial w _{j}} = \frac{1}{N} \sum_{i = 1}^{N} (\overset{y}{^}^{(i)} - y^{(i)}) x_{j}^{(i)}$

Derivation:

Single example ke liye, chain rule: $\frac{\partial L}{\partial w _{j}} = \frac{\partial L}{\partial y ^} \cdot \frac{\partial y ^}{\partial z} \cdot \frac{\partial z}{\partial w _{j}}$

Step 1: Prediction ke w.r.t. loss derivative $\frac{\partial L}{\partial y ^} = - \frac{y}{y ^} + \frac{1 - y}{1 - y ^}$

Kyun? $- y lo g \overset{y}{^}$ ka derivative $- y / \overset{y}{^}$ hai, aur $- (1 - y) lo g (1 - \overset{y}{^})$ ka derivative $(1 - y) / (1 - \overset{y}{^})$ hai.

Step 2: Sigmoid derivative (pehle derive kar chuke hain) $\frac{\partial y ^}{\partial z} = \overset{y}{^} (1 - \overset{y}{^})$

Step 3: Linear part ka derivative $\frac{\partial z}{\partial w _{j}} = x_{j}$

Combine karo: $\frac{\partial L}{\partial w _{j}} = [- \frac{y}{y ^} + \frac{1 - y}{1 - y ^}] \cdot \overset{y}{^} (1 - \overset{y}{^}) \cdot x_{j}$ $= [- y (1 - \overset{y}{^}) + (1 - y) \overset{y}{^}] x_{j}$ $= [- y + y \overset{y}{^} + \overset{y}{^} - y \overset{y}{^}] x_{j}$ $= (\overset{y}{^} - y) x_{j}$

Yeh step kyun? Cross-entropy aur sigmoid ke derivatives milkar complex fractions ko perfectly cancel kar dete hain, aur ek beautifully simple gradient bachta hai!

$N$ examples par average karo: $\frac{\partial J}{\partial w _{j}} = \frac{1}{N} \sum_{i = 1}^{N} (\overset{y}{^}^{(i)} - y^{(i)}) x_{j}^{(i)}$

Gradient descent update: $w_{j} := w_{j} - α \frac{\partial J}{\partial w _{j}}$

Worked Examples

Problem: Operating hours ke basis par predict karo ki ek turbine blade fail hoga ya nahi. Dataset:

$x^{(1)} = 1000$ hours, $y^{(1)} = 0$ (koi failure nahi)
$x^{(2)} = 5000$ hours, $y^{(2)} = 1$ (failure)

Initialize: $w = 0.0001$ , $b = - 0.3$ (features unnormalized hain illustration ke liye; chhota $w$ isliye choose kiya taaki $z$ ek sane range mein rahe).

Forward Pass (Example 2): $z^{(2)} = w \cdot x^{(2)} + b = 0.0001 \cdot 5000 - 0.3 = 0.5 - 0.3 = 0.2$ $\overset{y}{^}^{(2)} = σ (0.2) = \frac{1}{1 + e ^{- 0.2}} = \frac{1}{1.2214} = 0.5498$

Yeh step kyun? Hum pehle linear combination compute karte hain, phir probability paane ke liye sigmoid se pass karte hain. Note karo ki humne deliberately $w = 0.0001$ use kiya hai taaki $z$ sigmoid ko saturate na kare.

Is example ke liye Loss: $L^{(2)} = - [1 \cdot lo g (0.5498) + 0 \cdot lo g (0.4502)] = - lo g (0.5498) = 0.598$

Gradient: $\frac{\partial L ^{(2)}}{\partial w} = (\overset{y}{^}^{(2)} - y^{(2)}) \cdot x^{(2)} = (0.5498 - 1) \cdot 5000 = - 2251$

Negative kyun? Prediction bahut kam hai (0.55 vs 1), isliye $w$ increase karna hoga prediction badhane ke liye. Negative gradient matlab descent $w$ mein add karega.

Update ( $α = 1 0^{- 6}$ ): Kyunki raw feature ( $x = 5000$ ) bada hai, gradient bhi bada hai, isliye overshoot se bachne ke liye bahut chhota learning rate choose karte hain: $w := 0.0001 - 1 0^{- 6} \cdot (- 2251) = 0.0001 + 0.002251 = 0.002351$

Yeh step kyun? Unnormalized features ke saath gradient ka magnitude $x$ ke saath scale karta hai, isliye $α$ chhota hona chahiye. (Practice mein hum features normalize karte hain — Mistake 2 dekho — jisse hum ek bada, zyada stable $α$ use kar sakte hain.) Update ke baad $w$ increase hota hai, jiski wajah se model high operating hours ke liye zyada probability predict karta hai.

Problem: Cloud presence ke liye binary classifier. Features: average pixel brightness ( $x_{1}$ ), brightness variance ( $x_{2}$ ).

Training point: $x = [180, 45]$ , $y = 1$ (cloudy) Parameters: $w = [0.01, - 0.02]$ , $b = - 1.5$

Prediction: $z = 0.01 (180) + (- 0.02) (45) - 1.5 = 1.8 - 0.9 - 1.5 = - 0.6$ $\overset{y}{^} = σ (- 0.6) = \frac{1}{1 + e ^{0.6}} = \frac{1}{1 + 1.8221} = \frac{1}{2.8221} = 0.3543$

Yeh step kyun? Negative $z$ matlab model sochta hai "cloudy nahi" (prob < 0.5), lekin sach yeh hai ki cloudy hai ( $y = 1$ ) — error!

Loss: $L = - [1 \cdot lo g (0.3543)] = 1.0376$

Gradients: $\frac{\partial L}{\partial w _{1}} = (\overset{y}{^} - y) x_{1} = (0.3543 - 1) (180) = - 116.23$ $\frac{\partial L}{\partial w _{2}} = (0.3543 - 1) (45) = - 29.06$

Yeh step kyun? Dono gradients negative hain kyunki prediction bahut kam hai. Updates dono weights ko increase karenge taaki model in feature values ke liye zyada "pro-cloudy" bane.

Common Mistakes

Galat Soch: "Sigmoid 0.7 de raha hai, toh output 0.7 hai?"

Kyun Sahi Lagta Hai: Sigmoid ek number output karta hai, toh directly use karo.

The Fix: $\overset{y}{^}$ ek probability hai, final class nahi. Threshold apply karo:

prediction_class = 1 if y_hat >= 0.5 else 0

Aerospace safety applications ke liye, threshold 0.3 use kar sakte ho (missed failures ke bajaye false alarms prefer karo).

Yeh Kyun Kaam Karta Hai: Probabilities cost/benefit ke basis par decisions inform karti hain. 0.7 probability ka matlab "class 1 ka 70% jawab hai" nahi — iska matlab hai "class 1 mein 70% confidence hai."

Galat Code:

# x1 in [0, 1000] hours, x2 in [0, 1] boolean
z = w1 * x1 + w2 * x2 + b

Kyun Fail Hota Hai: $w_{1}$ dominate karega kyunki $x_{1}$ ki scale bahut badi hai. Gradient updates unstable ho jaate hain aur tumhe bahut chhota learning rate use karna padta hai (jaisa Example 1 mein dekha).

The Fix: Features standardize karo: $x_{j}^{norm} = \frac{x _{j} - μ _{j}}{σ _{j}}$

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

Yeh Kyun Kaam Karta Hai: Saare features equally contribute karte hain. Gradients ki magnitudes similar hoti hain, optimization faster converge hoti hai aur ek bada, stable learning rate use kar sakte ho.

Galat Math: $L = - [y lo g y]$ compute karna, $L = - [y lo g \overset{y}{^}]$ ke bajaye

Kyun Hota Hai: True label $y$ aur predicted probability $\overset{y}{^}$ mein confusion.

The Fix: Cross-entropy inn dono ke beech divergence measure karta hai:

True distribution: $y$ (ya toh 0 ya 1)
Predicted distribution: $\overset{y}{^}$ (sigmoid se probability)

Hamesha: $L = - [y lo g \overset{y}{^} + (1 - y) lo g (1 - \overset{y}{^})]$

Study Aids

Cross-Entropy: "Confident Wrongs ko Punish karo" $- lo g$ bahut bada ho jaata hai jab probability → 0. Agar model confident hai (high $\overset{y}{^}$ ) lekin galat hai ( $y = 0$ ), toh loss explode kar jaata hai.

Recall

Feynman Explanation (12 Saal ke Bachche ko Samjhao) Socho tum guess kar rahe ho ki coin heads hai ya tails, sirf yeh dekh kar ki woh kitna shiny hai. Tum directly "haan" ya "nahi" nahi bol sakte — tumhe batana hoga ki tum KITNE SURE ho, jaise "70% sure ki heads hai."

Sigmoid function ek special ruler ki tarah hai jo kisi bhi measurement (shine level) ko 0% aur 100% ke beech ek percentage mein badal deta hai. Agar shine bahut zyada hai, sigmoid kehta hai "almost 100% heads." Agar shine bahut kam hai, sigmoid kehta hai "almost 0% heads" (toh tails!). Agar shine medium hai, sigmoid kehta hai "50% — dono ho sakta hai."

Ab, computer kaise seekhta hai ki "zyada" aur "kam" shine ka kya matlab hai? Hum computer ko bahut saare coins aur unke sahi answers dikhate hain. Cross-entropy ek "wrongness score" hai — agar tumne kaha "90% heads" lekin woh tails tha, toh tumhe BAHUT BADA penalty milti hai. Agar tumne kaha "51% heads" aur woh heads tha, toh chhoti si penalty. Computer apna ruler adjust karta hai taaki wrongness score jitna ho sake utna chhota ho.

Aerospace mein, coins ki jagah hum guess karte hain "kya yeh engine part toot jaayega?" temperature aur vibration ke basis par. Same math!

Connections

Linear Regression: Logistic regression linear combination $z = w^{T} x + b$ use karta hai, lekin classification ke liye sigmoid add karta hai
Gradient Descent: Weight updates yahaan derive kiya hua $\nabla J$ use karte hain; same optimization algorithm
Softmax Regression: Multi-class generalization — $K > 2$ classes ke liye sigmoid, softmax ban jaata hai
Neural Networks: Sigmoid ek activation function hai; cross-entropy standard classification loss hai
Maximum Likelihood Estimation: Cross-entropy loss, Bernoulli model ka negative log-likelihood hai
ROC Curves and AUC: Decision threshold vary karke logistic regression evaluate karo
Regularization (L1, L2): Overfitting rokne ke liye loss function mein $λ ∥ w ∥$ add karo
Bayesian Interpretation: Logistic regression directly $P (y = 1∣ x)$ approximate karta hai

Flashcards

#flashcards/coding

Sigmoid function kya hai aur logistic regression mein kyun use hoti hai?

σ (z) = \frac{1}{1 + e ^{- z}}

. Yeh kisi bhi real-valued linear combination ko

(0, 1)

mein ek probability mein map karta hai, har jagah differentiable hai, aur iska derivative

σ (z) (1 - σ (z))

ka ek simple form hai jo gradient calculations ko efficient banata hai.

Log-odds assumption se sigmoid function derive karo

Maano

lo g (\frac{p}{1 - p}) = z

. Tab

\frac{p}{1 - p} = e^{z}

, isliye

p = e^{z} (1 - p)

, jisse

p (1 + e^{z}) = e^{z}

milta hai, yani

p = \frac{e ^{z}}{1 + e ^{z}} = \frac{1}{1 + e ^{- z}}

Single example ke liye binary cross-entropy loss kya hai?

L (y, \overset{y}{^}) = - [y lo g (\overset{y}{^}) + (1 - y) lo g (1 - \overset{y}{^})]

jahaan

y \in {0, 1}

true label hai aur

\overset{y}{^} \in (0, 1)

predicted probability hai.

Logistic regression mein MSE ki jagah cross-entropy loss kyun use hoti hai?

Sigmoid ke saath MSE non-convex hota hai (kai local minima) aur predictions saturate hone par vanishing gradients hote hain. Cross-entropy convex hai, maximum likelihood se derive hoti hai, aur confident galat predictions ke liye bhi strong gradients maintain karti hai.

Weight

w_{j}

ke w.r.t. cross-entropy loss ka gradient derive karo

Chain rule use karke:

\frac{\partial L}{\partial w _{j}} = \frac{\partial L}{\partial y ^} \cdot \frac{\partial y ^}{\partial z} \cdot \frac{\partial z}{\partial w _{j}} = [- \frac{y}{y ^} + \frac{1 - y}{1 - y ^}] \cdot \overset{y}{^} (1 - \overset{y}{^}) \cdot x_{j} = (\overset{y}{^} - y) x_{j}

Logistic regression mein decision boundary kya hoti hai?

Woh hyperplane jahaan

w^{T} x + b = 0

hota hai, jo

\overset{y}{^} = σ (0) = 0.5

correspond karta hai. Ek taraf ke points class 1 mein classify hote hain, doosri taraf class 0 mein.

Sigmoid function ka derivative kya hai?

σ^{'} (z) = σ (z) (1 - σ (z))

. Yeh quotient/chain rule se derive hota hai aur backpropagation ko efficient banata hai kyunki ise sirf sigmoid output chahiye, exponential dobara compute karne ki zarurat nahi.

Logistic regression mein features normalize kyun karne chahiye?

Bahut alag scales wale features kuch weights ko dominate karaate hain aur chhota learning rate use karne par majboor karte hain. Standardization (zero mean, unit variance) ensure karta hai ki saare features equally contribute karein aur gradients ki magnitudes similar hon, jisse convergence fast hoti hai.

Logistic regression aur maximum likelihood estimation mein kya relation hai?

Cross-entropy loss, Bernoulli distribution ka negative log-likelihood hai. Loss minimize karna = observed labels ki likelihood, given model parameters, maximize karna.

Aerospace safety-critical classification ke liye kaunsa threshold use karna chahiye?

Hamesha 0.5 nahi. Failure prediction ke liye, chhota threshold use karo (jaise 0.3) taaki missed failures ke bajaye false alarms prefer ho. Threshold cost asymmetry par depend karta hai:

cost (miss) ≫ cost (false alarm)