Exercises — Logistic regression — sigmoid, cross-entropy loss

Solution 1.1 Kya karte hain: $z=0$ ko $\sigma (z)=\frac{1}{1+e^{-z}}$ mein plug karo. $\sigma (0)=\frac{1}{1+e^0}=\frac{1}{1+1}=\frac{1}{2}=0.5$ Kaise dikhta hai: S-curve ka bilkul middle — woh point jo figure mein hai jahan curve dotted 0.5 line cross karti hai. Decision: $\hat{y}=0.5\ge 0.5$, toh hum class 1 predict karte hain (boundary hamare convention ke hisaab se inclusive hai).

Solution 1.2

• $N$ = training examples ki woh number jis par hum average karte hain. Average karna (sirf sum nahi) matlab yeh hai ki loss ka size sirf isliye nahi badhega kyunki humne zyada data collect kiya.
• $y^{(i)}$ = example $i$ ka true label, jo ya toh $0$ hai ya $1$ (beech mein kabhi nahi).
• ${\hat{y}}^{(i)}$ = model ki predicted probability ki example $i$ class 1 hai, ek number $(0,1)$ mein.
• Minus sign "log-likelihood jo hum maximise karna chahte hain" ko "loss jo hum minimise karna chahte hain" mein flip karta hai. Optimisers convention ke hisaab se neeche jaate hain, toh hum unhe kuch aisa dete hain jo fit improve hone par chhota hota jaata hai.

Fill-in check $\sigma (z)\to 1$ as $z\to$ ::: $+\infty$ $\sigma (z)\to 0$ as $z\to$ ::: $-\infty$ Woh value of $z$ jahan $\sigma$ exactly $0.5$ ke barabar hai ::: $z=0$

Solution 2.1 Step 1 — logit (dot product plus bias). Dot product kyun? Kyunki ${\mathbf{w}}^T\mathbf{x}$ woh single number hai jo kehta hai "features, importance se weight karke, class 1 ki taraf kitna push kar rahe hain." $z={\mathbf{w}}^T\mathbf{x}+b=(0.5)(2)+(-1)(3)+1=1-3+1=-1$ Step 2 — squash karo. $\hat{y}=\sigma (-1)=\frac{1}{1+e^1}=\frac{1}{1+2.71828}=\frac{1}{3.71828}=0.2689$ Kaise dikhta hai: $z=-1$ sigmoid figure mein centre ke left mein hai, toh hum 0.5 se neeche lagte hain. Decision: $0.2689<0.5\Rightarrow$ class 0.

Solution 2.2 Kya karte hain: $L=-[y\log \hat{y}+(1-y)\log (1-\hat{y})]$ use karo. $y=1$ ke saath doosra term khatam ho jaata hai ($1-y=0$), sirf $-\log \hat{y}$ bachta hai. $L=-\log (0.2689)=1.3133$ Itna bada kyun? Truth class 1 tha lekin model ne sirf $0.27$ probability di — ek confident galat lean. Cross-entropy isko punish karta hai. (Yahan logarithms natural logs hain, base $e$, jo sigmoid ke andar wale $e$ se match karta hai.)

Solution 2.3 Step 1 — error term. $\hat{y}-y=0.2689-1=-0.7311$. Negative kyunki prediction bahut kam thi. Step 2 — per-weight gradients. $\frac{\partial L}{\partial w_1}=(-0.7311)(2)=-1.4622,\frac{\partial L}{\partial w_2}=(-0.7311)(3)=-2.1933$ Step 3 — descend ($w_j:=w_j-\alpha \partial L/\partial w_j$): $w_1:=0.5-0.1(-1.4622)=0.6462,w_2:=-1-0.1(-2.1933)=-0.7807$ Kaise dikhta hai: dono weights is tarah move kiye ki agली prediction raise ho — exactly waही jo hum chahte hain kyunki truth class 1 hai. Yeh Gradient Descent ka ek step hai.

Solution 3.1 Compute karo: $(\hat{y}-y)x_j=(1-1)x_j=0$ har feature $j$ ke liye. Kyun important hai: update $w_j:=w_j-\alpha \cdot 0=w_j$ weight ko unchanged chhod deta hai. Ek perfectly-classified point aage ki learning mein kuch contribute nahi karta — algorithm pehle se "use kar chuka hai" jo woh point sikha sakta tha. Yeh minimum ka geometric meaning hai: koi slope nahi, koi motion nahi. Subtle baat: $\hat{y}=1$ ke liye $z\to +\infty$ chahiye, jo finite weights se reachable nahi hai. Toh practice mein gradient zero ki taraf shrink hota hai lekin truly kabhi hit nahi karta — model weights ko badhata rehta hai. Woh endless drift exactly woh problem hai jiske liye Regularization (L1, L2) exist karta hai.

Solution 3.2 Losses: $L_a=-\log (0.5)=0.6931,L_b=-\log (0.01)=4.6052$ Error terms: $(\hat{y}-y{)}_a=0.5-1=-0.5,(\hat{y}-y{)}_b=0.01-1=-0.99$ Padhna: confidently-wrong case (b) mein bahut bada loss aur bada-magnitude error term dono hain, toh yeh zyada strong corrective push produce karta hai. Cross-entropy ka squared error ke upar yahi poora point hai: confident hona aur galat hona bahut zyada hurt karta hai, jo model ko us bure state se jaldi bahar kheenchta hai. Ek learner jo sirf unsure hai use gentle nudge milta hai.

Solution 4.1 Forward pass, point 1 ($z^{(1)}=1\cdot 0.5+0=0.5$): ${\hat{y}}^{(1)}=\sigma (0.5)=\frac{1}{1+e^{-0.5}}=0.6225$ Forward pass, point 2 ($z^{(2)}=1\cdot 2.0+0=2.0$): ${\hat{y}}^{(2)}=\sigma (2.0)=\frac{1}{1+e^{-2.0}}=0.8808$ Error terms: $(0.6225-0)=0.6225$; $(0.8808-1)=-0.1192$. Averaged gradients ($N=2$): $\frac{\partial J}{\partial w}=\frac{1}{2}[(0.6225)(0.5)+(-0.1192)(2.0)]=\frac{1}{2}[0.3112-0.2384]=0.0364$ $\frac{\partial J}{\partial b}=\frac{1}{2}[0.6225+(-0.1192)]=\frac{1}{2}(0.5034)=0.2517$ Update: $w:=1.0-0.5(0.0364)=0.9818,b:=0.0-0.5(0.2517)=-0.1258$ Kaise dikhta hai / sanity: point 1 (ek non-failure jise model ne over-predict kiya) $w$ ko neeche push karta hai; point 2 (ek failure jise usne thoda under-predict kiya) $w$ ko upar push karta hai. Woh $w$ par nearly cancel ho jaate hain, lekin dono $b$ ko lower karne par agree karte hain — poori S-curve ko rightward shift karte hue taaki $0.5$ ke scores "fail hone ki kam likelihood" padhein. Ek coherent, honest step.

Solution 4.2 Key link: $\hat{y}=0.5⟺\sigma (z)=0.5⟺z=0$. Toh boundary wahan hai jahan logit exactly zero hai — yahi reason hai ki logistic regression ki boundary hamesha ek straight line (ya hyperplane) hoti hai, seedha Linear Regression se inherited. $z=0$ set karo: $2x_1-1\cdot x_2-1=0⟹x_2=2x_1-1$ Description: slope $2$ aur intercept $-1$ wali ek straight line. Ek taraf $z>0$ (class 1 predict karo), doosri taraf $z<0$ (class 0 predict karo). Sigmoid sirf decide karta hai ki us line ko cross karte waqt probability kitni tez turn karti hai, nahi ki line kahan hai.

Solution 5.1 Definition se shuru karo aur $-z$ substitute karo: $\sigma (-z)=\frac{1}{1+e^{-(-z)}}=\frac{1}{1+e^z}$ Top aur bottom ko $e^{-z}$ se multiply karo: $=\frac{e^{-z}}{e^{-z}+1}=\frac{e^{-z}+1-1}{1+e^{-z}}=\frac{1+e^{-z}}{1+e^{-z}}-\frac{1}{1+e^{-z}}=1-\sigma (z)$ Interpretation: score $z$ par "not class 1" ki probability, flipped score $-z$ par "class 1" ki probability ke barabar hai. Curve mein $(0,0.5)$ ke baare mein point symmetry hai. Exactly yahi reason hai ki ek two-class problem ko sirf ek sigmoid chahiye: $P(\text{class }1)$ model karna automatically $P(\text{class }0)=1-\hat{y}$ pin kar deta hai. Jab do se zyada classes hoon toh yeh freebie khatam ho jaati hai aur tumhe Softmax Regression ki taraf move karna padta hai.

Solution 5.2 First derivative — do known facts chain karo $\frac{dL}{d\hat{y}}=-\frac{y}{\hat{y}}+\frac{1-y}{1-\hat{y}}$ aur $\frac{d\hat{y}}{dz}=\hat{y}(1-\hat{y})$ ($\hat{y}=\sigma (z)$ ke saath): $\frac{dL}{dz}=[-\frac{y}{\hat{y}}+\frac{1-y}{1-\hat{y}}]\hat{y}(1-\hat{y})=-y(1-\hat{y})+(1-y)\hat{y}=\hat{y}-y=\sigma (z)-y$ Second derivative — $\sigma (z)-y$ differentiate karo; constant $y$ khatam ho jaata hai aur ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$: $\frac{d^{2}L}{d{z}^2}=\sigma (z)(1-\sigma (z))$ Sign: $\sigma (z)\in (0,1)$, toh dono factors strictly positive hain, toh second derivative har jagah strictly positive hai. Conclusion: ek function jiska har jagah positive second derivative ho woh convex hai — yeh ek bowl ki tarah upar curve karta hai, ek single global minimum guarantee karta hai bina kisi false valleys ke jo Gradient Descent ko trap kar sakein. Yeh rigorous reason hai ki humne squared error kyun chhhoda, aur yeh parent note mein Maximum Likelihood Estimation derivation ka promised payoff hai.

Solution 5.3 $\sigma (0)=0.5$. First derivative: $\frac{dL}{dz}=\sigma (0)-y=0.5-1=-0.5$ (negative: $z=0$ par loss abhi bhi decrease hoti hai jab hum $z$ ko correct class 1 ki taraf push karte hain — good). Second derivative: $\sigma (0)(1-\sigma (0))=0.5\times 0.5=0.25>0$ (positive: bowl upar curve karta hai, is point par convexity confirm karta hai).

Poori page ke liye Mnemonic Woh ek identity jo yahan har exercise unlock karti hai ::: gradient $(\hat{y}-y)x_j$ tak collapse ho jaata hai — "prediction minus truth, feature se scale karke."