Logistic regression — sigmoid, cross-entropy loss: Question bank

5.6.2 · D5 · Coding › Machine Learning (Aerospace Applications) › Logistic regression — sigmoid, cross-entropy loss

Yeh bank un misconceptions ko pakadta hai jo logistic regression mein hoti hain — sigmoid ka kya matlab hai, cross-entropy kyun use karte hain naa ki squared error, "linear" kiska refer karta hai, aur boundaries par kya hota hai (zero probability, perfect separation, saturated logits). Question padhon, apna jawab ek poore sentence mein bol ke dekho, phir reveal karo.

True or false — justify

Logistic regression ek regression algorithm hai, classification nahi.

Practice mein False hai — yeh sigmoid ke zariye ek continuous probability output karta hai (toh naam mein "regression" log-odds model karne ko refer karta hai), lekin hum us probability ko threshold karke classify karte hain, isliye isse classifier ki tarah use kiya jaata hai.

Logistic regression ki decision boundary curved hoti hai kyunki sigmoid curved hai.

False — boundary wahan hoti hai jahan

\overset{y}{^} = 0.5

ho, yaani jahan

z = w^{T} x + b = 0

ho, jo ek straight line (hyperplane) hai. Sigmoid sirf probability ko us flat boundary ke upar bend karta hai, boundary ko khud nahi.

Cross-entropy loss ek real trained model ke liye exactly zero ho sakta hai.

False — zero loss ke liye

y = 1

example par

\overset{y}{^} = 1

chahiye, lekin

σ (z) = 1

sirf tab hota hai jab

z \to \infty

. Aap zero loss ke kareeb aa sakte ho lekin finite weights se kabhi reach nahi kar sakte.

Agar aap har weight aur bias ko double kar do, toh classifications waisi hi rahegi.

Hard labels ke liye True —

z

ko positive constant se scale karna

z = 0

boundary ko nahi hilata, toh har point kis side par hai yeh unchanged rehta hai; lekin probabilities zyada extreme ho jaati hain (zyada confident).

Mean squared error bhi logistic regression train karne ke liye theek kaam karega.

Ek meaningful sense mein False — sigmoid ke saath MSE non-convex hai (local minima hain) aur jab sigmoid saturate karta hai toh iska gradient vanish ho jaata hai, toh confident-wrong points par training ruk jaati hai; cross-entropy convex hai aur wahan bhi strong gradient maintain karta hai.

0.5

ka sigmoid output matlab hai model confident hai ki class exactly halfway hai.

False —

0.5

matlab maximal uncertainty hai (model ke paas kisi bhi class ko favor karne ki koi information nahi), naa ki "half class." Yeh exactly

z = 0

ki decision boundary par baitha hai.

Cross-entropy predictions

\overset{y}{^}

mein aur raw weights

w

mein bhi convex hai.

Logistic regression ke liye specifically True — kyunki

\overset{y}{^} = σ (w^{T} x)

convex loss ko sigmoid ke saath is tarah compose karta hai ki overall objective

w

mein convex rehta hai, ek single global minimum guarantee karta hai.

Spot the error

Ek student

y = 1

ke liye loss

L = - lo g (1 - \overset{y}{^})

likhta hai. Kya galat hai?

Unhone

y = 0

wali branch use ki.

y = 1

ke liye loss

- lo g (\overset{y}{^})

hoti hai;

- lo g (1 - \overset{y}{^})

use karna

\overset{y}{^} \to 0

ko reward karta hai, jo ulta hai.

Koi claim karta hai gradient

\frac{\partial L}{\partial w _{j}} = (y - \overset{y}{^}) x_{j}

hai. Theek karo.

Sign ulta hai — sahi gradient

(\overset{y}{^} - y) x_{j}

hai. Unka version uphill point karta hai, toh gradient descent loss ko minimize karne ki jagah climb karega.

Ek note mein likha hai "log-odds

σ (w^{T} x + b)

ke barabar hain." Galti kahan hai?

Log-odds seedha

z = w^{T} x + b

ke barabar hain (linear); probability

σ (z)

ke barabar hoti hai. Sigmoid log-odds ko → probability mein map karta hai, yeh log-odds produce nahi karta.

Code loss ko -y*log(y_hat) - (1-y)*log(1-y_hat) compute karta hai aur kabhi kabhi nan return karta hai. Kyun?

Jab

\overset{y}{^}

exactly

0

1

hit karta hai,

lo g (0) = - \infty

NaN produce karta hai. Real code mein

\overset{y}{^}

[ϵ, 1 - ϵ]

mein clip kiya jaata hai isse avoid karne ke liye.

Ek student kehta hai "kyunki

σ^{'} (z) = σ (z) (1 - σ (z))

, weights ke w.r.t. gradient mein bhi

\overset{y}{^} (1 - \overset{y}{^})

factor hoga." Sahi hai?

False — woh factor cross-entropy derivative

- \frac{y}{y ^} + \frac{1 - y}{1 - y ^}

se cancel ho jaata hai, aur clean

(\overset{y}{^} - y) x_{j}

bach jaata hai. Yeh cancellation hi poora reason hai ki hum sigmoid ko cross-entropy ke saath pair karte hain.

Koi

\overset{y}{^}

\overset{y}{^} + (1 - \overset{y}{^})

se divide karke do classes ke across probabilities normalize karta hai. Bekaar hai?

Haan —

\overset{y}{^} + (1 - \overset{y}{^}) = 1

already hai, toh division kuch nahi karta. Binary logistic regression already normalized hai kyunki

1 - \overset{y}{^}

construction se hi class-0 probability hai.

Why questions

Log-odds ko directly probability ki jagah linear kyun model karte hain?

Kyunki probability

[0, 1]

mein rehti hai lekin linear function

w^{T} x + b

saare reals par range karta hai — log-odds bhi saare reals par range karta hai, toh wahan linearity consistent hai; sigmoid exactly woh function hai jo log-odds ko undo karta hai.

Negative gradient increasing

w

ki taraf kyun point karta hai jab

\overset{y}{^} < y

ho?

Kyunki gradient

(\overset{y}{^} - y) x_{j}

hai; agar

\overset{y}{^} < y

(aur

x_{j} > 0

) toh yeh negative hai, aur descent ek negative subtract karta hai, toh

w_{j}

badhta hai —

z

ko upar push karta hai aur

\overset{y}{^}

ko true label ki taraf le jaata hai.

Hum likelihood maximize kyun karte hain lekin isse loss minimize kyun bolte hain?

Cross-entropy data par average ki gayi negative log-likelihood hai; likelihood maximize karna aur uski negative log minimize karna same optimization hai, hum sirf sign flip karte hain taaki gradient descent (ek minimizer) apply ho sake. Dekho Maximum Likelihood Estimation.

Cross-entropy ek confident wrong prediction ko hesitant wale se itna zyada kyun penalize karta hai?

Kyunki

- lo g (\overset{y}{^}) \to \infty

jab

\overset{y}{^} \to 0

; galat class ke baare mein 99% sure hona near-infinite "surprise" lagata hai, jabki 55% wrong hona sirf thoda cost karta hai.

Hum smooth sigmoid ki jagah sirf step function use kyun nahi kar sakte?

Step function ka derivative almost everywhere zero hota hai aur boundary par undefined jump hota hai, toh gradient descent ke paas follow karne ke liye koi slope nahi hoti; sigmoid har jagah smooth, non-zero gradient deta hai.

L2 regularization kabhi kabhi zaruri kyun hoti hai jab classes perfectly separable hon?

Perfect separation ke saath weights bina bound ke badh sakte hain (loss forever shrink karne ke liye

\overset{y}{^}

ko exactly 0/1 ki taraf push karte hain), toh training kabhi converge nahi karti; Regularization (L1, L2) ek penalty add karta hai jo weights ko finite rakhta hai.

Zyada features add karne par boundary generally hyperplane kyun rehti hai, curve nahi?

Kyunki

z

features mein linear rehta hai jo aap usse feed karte ho; curved boundary tabhi aati hai jab aap nonlinear features engineer karo (jaise

x^{2}

) ya layers stack karo jaise Neural Networks mein.

Edge cases

z = 0

hone par

\overset{y}{^}

kya hota hai, aur iska kya matlab hai?

σ (0) = 0.5

exactly — model decision boundary par hai maximal uncertainty ke saath, kisi bhi class ko favor nahi kar raha.

Jab

z \to + \infty

aur

z \to - \infty

\overset{y}{^}

aur sigmoid ke slope ka kya hota hai?

\overset{y}{^} \to 1

aur

\overset{y}{^} \to 0

respectively, jabki slope

\overset{y}{^} (1 - \overset{y}{^}) \to 0

dono limits mein — sigmoid saturate ho jaata hai aur extreme logits ke liye learning slow ho jaati hai.

Agar ek example ke saare features zero hain, toh

\overset{y}{^}

kya hai?

z = w^{T} 0 + b = b

, toh

\overset{y}{^} = σ (b)

— prediction sirf bias par depend karti hai, jo class 1 ki base rate encode karta hai.

Loss gradient

(\overset{y}{^} - y) x_{j}

ka kya hota hai jab

x_{j} = 0

ho?

Woh weight ke liye zero hota hai — ek feature jo is example ke liye zero hai woh apne weight ko update karne ke liye koi signal nahi deta, chahe prediction error kuch bhi ho.

Ek dataset mein ek class 99% time occur karti hai. Ek untrained/lazy logistic model kya karta hai?

Uninformative features ke saath, yeh ek bada

b

seekhta hai taaki

σ (b) \approx 0.99

ho, hamesha majority class predict karta hai — high accuracy lekin useless recall, isliye raw accuracy ki jagah ROC Curves and AUC matter karta hai.

Agar do identical feature vectors ke opposite labels hain, kya training loss zero kar sakti hai?

Nahi — koi single

\overset{y}{^}

y = 1

aur ek

y = 0

example dono satisfy nahi kar sakta, toh model us probability par settle karta hai jo dono losses balance kare (close to

0.5

), aur residual loss irreducible hoti hai.

Agar training data mein exactly ek class hai toh logistic regression kya ban jaata hai?

Likelihood maximize hoti hai har point ke liye

\overset{y}{^}

ko us class ki taraf push karke, toh weights/bias diverge ho jaate hain — problem degenerate hai aur regularization ki zarurat hai ya setup ill-posed hai.

Binary logistic regression

K = 2

case of Softmax Regression se kaise relate karta hai?

Dono equivalent hain — do classes ke saath softmax sigmoid mein collapse ho jaata hai kyunki do class scores ko hamesha shift kiya ja sakta hai taaki ek zero ho, ek single logit bach jaata hai.

Recall Fast self-test

Gradient ki magic simplification kis do functions ko pair karne se aati hai? ::: Sigmoid aur cross-entropy — unke derivatives cancel ho jaate hain aur $(\overset{y}{^} - y) x_{j}$ bach jaata hai. Decision boundary kis equation se define hoti hai? ::: $z = w^{T} x + b = 0$ , equivalently $\overset{y}{^} = 0.5$ . Perfect separation plain (unregularized) logistic regression ko kyun break kar deta hai? ::: Weights ever-smaller loss chase karte hue bina bound ke badhte hain, toh training kabhi converge nahi karti.