Cost function (MSE) and gradient descent fitting

2.2.4 · AI-ML › Linear & Logistic Regression

Overview

Linear regression mein, hum apne data ke through best-fit line dhundna chahte hain. Lekin "best" ka matlab kya hai? Humein ek tarika chahiye jo measure kare ki hamare predictions kitne galat hain, aur phir unhe systematically improve kare. Mean Squared Error (MSE) hamare galat hone ka measure hai, aur gradient descent humara systematic improvement algorithm hai.

Hum absolute values ki jagah square isliye karte hain kyunki squared functions har jagah differentiable hote hain, jo hume calculus use karke minimum efficiently dhundhne deta hai.

Cost Function: Mean Squared Error

$J(\theta )=\frac{1}{2n}{\sum }_{i=1}^n(h_{\theta }(x_i)-y_i{)}^2$

Jahan:

• $J(\theta )$ = cost function (jo hum minimize karna chahte hain)
• $\theta$ = parameters (weights aur bias: ${\theta }_0,{\theta }_1,..$)
• $h_{\theta }(x_i)={\theta }_0+{\theta }_1{x}_i$ simple linear regression ke liye
• $\frac{1}{2}$ ek convenience factor hai jo derivatives lete waqt cancel ho jaata hai

First principles se derivation

Hume cost function ki zarurat kyun hai? Hamare paas $n$ training examples $(x_i,y_i)$ hain. Hamari hypothesis predictions generate karti hai ${\hat{y}}_i=h_{\theta }(x_i)$. Har prediction mein ek error hai: $e_i={\hat{y}}_i-y_i$.

Step 1: Total error measure karo Sum of errors: $\sum e_i$ kaam nahi karega (positive aur negative cancel ho jaate hain) Sum of absolute errors: $\sum |e_i|$ kaam karta hai lekin zero par differentiable nahi hai

Step 2: Errors ko square karo $\text{Total squared error}={\sum }_{i=1}^n{e}_i^2={\sum }_{i=1}^n(h_{\theta }(x_i)-y_i{)}^2$

Ye step kyun? Squaring se saare errors positive ho jaate hain, badi errors zyada penalize hoti hain, aur ek smooth surface banti hai jise hum calculus se optimize kar sakte hain.

Step 3: MSE paane ke liye average karo $J(\theta )=\frac{1}{n}{\sum }_{i=1}^n(h_{\theta }(x_i)-y_i{)}^2$

Ye step kyun? $n$ se divide karne par cost dataset size se independent ho jaati hai—5 size ki 10 errors ka cost same hona chahiye chahe hamare paas 100 examples hon ya 1000.

Step 4: $\frac{1}{2}$ factor add karo $J(\theta )=\frac{1}{2n}{\sum }_{i=1}^n(h_{\theta }(x_i)-y_i{)}^2$

Ye step kyun? Jab hum derivative lete hain, power rule squaring se 2 ka factor neeche laata hai, jo $\frac{1}{2}$ ke saath cancel ho jaata hai. Isse gradient cleaner ban jaata hai.

• Outer sum: Saare training examples ke errors accumulate karta hai
• Inner difference: Prediction minus actual (residual)
• Square: Errors ko quadratically penalize karta hai
• Division by 2n: Normalize karta hai aur derivatives simplify karta hai

Gradient Descent: Minimum Dhundna

Gradient $\nabla J(\theta )$ ek vector hai jo sabse steep increase ki direction mein point karta hai. Hum neeche utarne ke liye ulti direction mein jaate hain.

Gradient Descent Update Rule Derive karna

Step 1: Update formula likho (general form) ${\theta }_j:={\theta }_j-\alpha \frac{\partial J(\theta )}{\partial {\theta }_j}$

Jahan:

• $:=$ ka matlab hai "update to"
• $\alpha$ learning rate hai (step size)
• $\frac{\partial J(\theta )}{\partial {\theta }_j}$ parameter ${\theta }_j$ ke respect mein partial derivative (slope) hai

Ye form kyun? Agar slope positive hai (hum uphill par hain), derivative positive hai, toh hum use subtract karte hain (left/down jaate hain). Agar slope negative hai (hum downhill par hain), hum ek negative subtract karte hain (right/down jaate hain).

Step 2: Partial derivative compute karo Cost function se shuru karo: $J(\theta )=\frac{1}{2n}{\sum }_{i=1}^n(h_{\theta }(x_i)-y_i{)}^2$

Simple linear regression ke liye, $h_{\theta }(x_i)={\theta }_0+{\theta }_1{x}_i$.

${\theta }_j$ ke respect mein derivative lo: $\frac{\partial J(\theta )}{\partial {\theta }_j}=\frac{\partial }{\partial {\theta }_j}\left[\frac{1}{2n}{\sum }_{i=1}^n(h_{\theta }(x_i)-y_i{)}^2\right]$

Ye step kyun? Hume jaanna hai ki ${\theta }_j$ mein ek choti si change cost ko kitna change karti hai.

Step 3: Chain rule apply karo $=\frac{1}{2n}{\sum }_{i=1}^n\frac{\partial }{\partial {\theta }_j}(h_{\theta }(x_i)-y_i{)}^2$

$=\frac{1}{2n}{\sum }_{i=1}^{n}2(h_{\theta }(x_i)-y_i)\cdot \frac{\partial }{\partial {\theta }_j}(h_{\theta }(x_i)-y_i)$

Ye step kyun? Chain rule: $u^2$ ki derivative $2u\cdot u^{\prime }$ hoti hai. Yahan $u=h_{\theta }(x_i)-y_i$.

2, $\frac{1}{2}$ ke saath cancel ho jaata hai: $=\frac{1}{n}{\sum }_{i=1}^n(h_{\theta }(x_i)-y_i)\cdot \frac{\partial h_{\theta }(x_i)}{\partial {\theta }_j}$

Step 4: $\frac{\partial h_{\theta }(x_i)}{\partial {\theta }_j}$ compute karo

${\theta }_0$ ke liye (bias/intercept): $\frac{\partial }{\partial {\theta }_0}({\theta }_0+{\theta }_1{x}_i)=1$

${\theta }_1$ ke liye (slope): $\frac{\partial }{\partial {\theta }_1}({\theta }_0+{\theta }_1{x}_i)=x_i$

Generally, ${\theta }_j$ ke liye: $\frac{\partial h_{\theta }(x_i)}{\partial {\theta }_j}=x_i^{(j)}$ jahan $x_i^{(j)}$ example $i$ ka $j$-th feature hai (bias term ke liye $x_i^{(0)}=1$ ke saath).

Step 5: Final gradient $\frac{\partial J(\theta )}{\partial {\theta }_j}=\frac{1}{n}{\sum }_{i=1}^n(h_{\theta }(x_i)-y_i)\cdot x_i^{(j)}$

Saare parameters simultaneously update karo:

• ${\theta }_0:={\theta }_0-\alpha \frac{1}{n}{\sum }_{i=1}^n(h_{\theta }(x_i)-y_i)$
• ${\theta }_1:={\theta }_1-\alpha \frac{1}{n}{\sum }_{i=1}^n(h_{\theta }(x_i)-y_i)\cdot x_i$

Simultaneous ka matlab: purane $\theta$ values use karke saari nayi values compute karo, phir sab ek saath update karo.

Learning Rate $\alpha$

Bahut chota: Convergence slow hoti hai (bahut saare iterations chahiye) Bahut bada: Hum minimum overshoot kar sakte hain aur diverge ho sakte hain (cost badhti hai) Bilkul sahi: Minimum tak fast convergence

Typical values: $0.001,0.01,0.1,1$. Aksar trial and error ya adaptive methods se milte hain.

Gradient tumhe direction batata hai; $\alpha$ batata hai ki kitna door jaana hai.

Worked Examples

Goal: $y={\theta }_0+{\theta }_{1}x$ ko gradient descent se fit karo.

Initialize: ${\theta }_0=0,{\theta }_1=0$, learning rate $\alpha =0.1$

Iteration 1:

1. Predictions compute karo:

   • $h_{\theta }(1)=0+0(1)=0$
   • $h_{\theta }(2)=0+0(2)=0$
   • $h_{\theta }(3)=0+0(3)=0$

2. Errors (residuals) compute karo:

   • $e_1=0-2=-2$
   • $e_2=0-4=-4$
   • $e_3=0-5=-5$

3. Cost compute karo: $J(\theta )=\frac{1}{2\cdot 3}[(-2{)}^2+(-4{)}^2+(-5{)}^2]=\frac{1}{6}[4+16+25]=\frac{45}{6}=7.5$

4. Gradients compute karo: $\frac{\partial J}{\partial {\theta }_0}=\frac{1}{3}[(-2)(1)+(-4)(1)+(-5)(1)]=\frac{-11}{3}=-3.67$ Ye step kyun? Har error ko bias term ke liye $x^{(0)}=1$ se multiply kiya jaata hai.

   $\frac{\partial J}{\partial {\theta }_1}=\frac{1}{3}[(-2)(1)+(-4)(2)+(-5)(3)]=\frac{-25}{3}=-8.33$

   Ye step kyun? Har error ko corresponding $x_i$ value se multiply kiya jaata hai.

5. Parameters update karo: ${\theta }_0:=0-0.1(-3.67)=0.367$ ${\theta }_1:=0-0.1(-8.33)=0.833$

   Ye step kyun? Hum gradient ki ulti direction mein jaate hain (downhill).

Iteration 2 (naye $\theta$ se):

1. Predictions:

   • $h_{\theta }(1)=0.367+0.833(1)=1.2$
   • $h_{\theta }(2)=0.367+0.833(2)=2.033$
   • $h_{\theta }(3)=0.367+0.833(3)=2.866$

2. Errors:

   • $e_1=1.2-2=-0.8$
   • $e_2=2.033-4=-1.967$
   • $e_3=2.866-5=-2.134$

3. Cost: $J(\theta )=\frac{1}{6}[0.64+3.87+4.55]=1.51$

   Ye step kyun? Cost 7.5 se 1.51 ho gayi—hum downhill move kar rahe hain!

Tab tak continue karo jab tak cost significantly decrease karna band na kar de (convergence).

Step 1: Prediction aur error compute karo

• $h_{\theta }(2)=1\cdot 2=2$
• Error: $e=2-5=-3$

Step 2: Cost compute karo $J({\theta }_1)=\frac{1}{2\cdot 1}(-3{)}^2=4.5$

Step 3: Gradient compute karo $\frac{\partial J}{\partial {\theta }_1}=\frac{1}{1}[(-3)(2)]=-6$

Ye negative kyun hai? Hamari prediction (2) actual (5) se kam hai. Error negative hai. $x=2$ se multiply karne par, hume -6 milta hai. Ye negative gradient batata hai: ${\theta }_1$ badhao taaki better predictions ho sakein.

Step 4: Update karo ($\alpha =0.1$ ke saath) ${\theta }_1:=1-0.1(-6)=1+0.6=1.6$

Ye step kyun? Negative gradient subtract karne ka matlab hai add karna, isliye ${\theta }_1$ badhta hai, jo hamare predictions badhayega (2 ko 5 ke kareeb le jaayega).

Verification:

• Nayi prediction: $h_{\theta }(2)=1.6\cdot 2=3.2$ (5 ke zyada kareeb ✓)
• Nayi cost: $J=\frac{1}{2}(3.2-5{)}^2=1.62$ (4.5 se kam ✓)

Common Mistakes

Ye sahi kyun lagta hai: Lagta hai most recent value use karna logical hai.

Ye galat kyun hai: Gradient compute karte waqt assume kiya gaya tha ki dono parameters apni purani values par hain. Partially updated value use karne se gradient direction corrupt ho jaati hai.

Fix: Pehle saare gradients compute karo, phir saare parameters ek saath update karo.

# Galat
theta_0 = theta_0 - alpha * grad_0
theta_1 = theta_1 - alpha * grad_1  # NEW theta_0 indirectly use karta hai
 
# Sahi
temp_0 = theta_0 - alpha * grad_0
temp_1 = theta_1 - alpha * grad_1
theta_0 = temp_0
theta_1 = temp_1

Ye sahi kyun lagta hai: Derivation mein sum dikhta hai, toh bas sum use karo.

Ye galat kyun hai: $n$ se divide kiye bina, tumhara effective learning rate dataset size ke saath scale karta hai. Zyada data add karne par $\alpha$ ko shrink karna padega, jo ise non-transferable bana deta hai.

Fix: Hamesha $n$ se divide karo: ${\theta }_j:={\theta }_j-\alpha \frac{1}{n}{\sum }_{i=1}^n(h_{\theta }(x_i)-y_i)\cdot x_i^{(j)}$

Ye sahi kyun lagta hai: Lagta hai saara data use karne se ye faster hoga.

Actually: Ye stochastic gradient descent (SGD) hai, batch gradient descent nahi. Ye galat per se nahi hai, lekin ye ek alag algorithm hai alag properties ke saath:

• Batch GD: Saara data use karta hai, stable convergence, iteration ke hisab se slower
• SGD: Ek point use karta hai, noisy updates, iteration ke hisab se faster, exactly converge nahi ho sakta

Dono valid hain; bas jaano ki tum kaunsa use kar rahe ho!

Ye kyun hota hai: Tum minimum overshoot kar rahe ho. Har update valley ke doosri taraf jump kar jaata hai.

Fix:

1. $\alpha$ kam karo (10 se divide karke try karo)
2. Iteration-wise cost monitor karo—batch GD mein ye monotonically decrease honi chahiye
3. Learning rate schedules ya adaptive methods use karo (Adam, RMSprop)

Visual: Agar cost vs. iteration plot jagged upward trend dikhaye, $\alpha$ bahut bada hai.

Convergence Criteria

Practice mein, cost function monitor karo. Agar ye meaningfully decrease karna band kar de, toh converge ho gaye.

Batch vs. Stochastic vs. Mini-Batch

Batch GD (jo humne derive kiya): ${\theta }_j:={\theta }_j-\alpha \frac{1}{n}{\sum }_{i=1}^n(h_{\theta }(x_i)-y_i)x_i^{(j)}$

Stochastic GD: ${\theta }_j:={\theta }_j-\alpha (h_{\theta }(x_i)-y_i)x_i^{(j)}\text{for a random }i$

Mini-batch GD: ${\theta }_j:={\theta }_j-\alpha \frac{1}{b}{\sum }_{i\in \text{batch}}(h_{\theta }(x_i)-y_i)x_i^{(j)}$

Feature Scaling ka Impact

Solution: Features ko similar scales par normalize karo (e.g., mean 0, std 1): $x_j^{(i)}:=\frac{x_j^{(i)}-{\mu }_j}{{\sigma }_j}$

Isse cost function zyada circular/spherical ban jaata hai, jo gradient descent ko bade learning rates ke saath faster converge karne deta hai.

Ab, best line kaise dhundte hain? Ek random line se shuru karo. Phir dekho ki cost ko chhota karne ke liye use kis taraf tilt karna chahiye—jaise ball ko pahaad se neeche dhakka dena. Use thoda si us direction mein tilt karo. Phir dobara check karo aur aur tilt karo. Ye tab tak karte raho jab tak tumhari line itni achi na ho jaaye ki use aur tilt karna koi help nahi kare. Yahi gradient descent hai—ye tumhari line ko sikhata hai ki valley ke bottom tak pahunchne ke liye downhill walk karo, jahan cost sabse kam ho.

"Learning rate" batata hai tumhare kadam kitne bade hain. Bahut bade aur tum valley ke upar se jump kar jaoge. Bahut chote aur pahunchne mein forever lag jaayega.

• Measure: $J(\theta )$ compute karo = tum kitne galat ho
• Squared: Errors squared hain (badi galtiyan zyada hurt karti hain)
• Errors: Prediction aur actual ke beech difference
• Go: Parameters ko ek direction mein move karo
• Downhill: Gradient ki ulti direction (steepest descent)

Ya socho: "Mean Squared Errors Guide Descent"

Connections

• 2.2.03-Hypothesis-representation-in-linear-regression — Hypothesis $h_{\theta }(x)$ jo hum optimize kar rahe hain
• 2.2.05-Normal-equation — Gradient descent ka closed-form alternative
• 2.06-Feature-scaling-and-normalization — Gradient descent ki convergence faster banata hai
• 2.3.02-Logistic-cost-function — Classification ke liye alag cost function
• 2.4.01-Gradient-descent-for-neural-networks — Backpropagation is idea ko extend karta hai
• 3.1.02-Convex-optimization — MSE convex hai, global minimum guarantee karta hai
• 4.2.01-Stochastic-gradient-descent — Large datasets ke liye faster variant
• 4.2.03-Learning-rate-schedules — $\alpha$ ke liye adaptive strategies

#flashcards/ai-ml

Linear regression ke liye Mean Squared Error (MSE) cost function kya hai? :: $J(\theta )=\frac{1}{2n}{\sum }_{i=1}^n(h_{\theta }(x_i)-y_i{)}^2$, jahan $h_{\theta }(x_i)$ prediction hai aur $y_i$ actual value. $\frac{1}{2n}$ dataset size se normalize karta hai aur derivatives simplify karta hai.

Parameter ${\theta }_j$ ke liye gradient descent update rule kya hai? :: ${\theta }_j:={\theta }_j-\alpha \frac{\partial J(\theta )}{\partial {\theta }_j}$, jahan $\alpha$ learning rate hai aur partial derivative us parameter ke respect mein cost function ka gradient (slope) hai.

Gradient descent mein negative gradient ka kya matlab hai? :: Negative gradient ka matlab hai ki parameter badhne par cost decrease hoti hai, isliye hume parameter increase karna chahiye. Kyunki hum gradient subtract karte hain ($\theta :=\theta -\alpha \cdot \text{negative value}$), hum effectively $\theta$ mein add karte hain, use us direction mein move karte hain jo cost reduce karta hai.

Concept Map