Foundations — Linear regression — normal equation, gradient descent derivation

5.6.1 · D1 · Coding › Machine Learning (Aerospace Applications) › Linear regression — normal equation, gradient descent deriva

Tumhe parent note Linear Regression fluently padhne se pehle, us mein aane wala har ek symbol apna banana hoga. Yeh page har ek symbol ko scratch se build karta hai, ek aisi order mein jahan har block usse pehle waale par tikaa ho. Upar se neeche padho.

1. Ek data point $(x_i,y_i)$

Socho dots ki ek scatter. Har dot ek horizontal position $x_i$ aur ek height $y_i$ par baithi hai.

• $x_i$ → dot kitni right mein hai (woh cheez jo tum jaante ho, jaise altitude).
• $y_i$ → dot kitni upar hai (woh cheez jo tum predict karna chahte ho, jaise fuel burn).
• $n$ → dots ki count. Agar $i$ runs $1,2,3$ karta hai toh $n=3$.

Yeh topic isko kyun chahta hai. Regression ka poora kaam in sab dots ko ek rule se summarise karna hai. "Dot number $i$" ke liye koi symbol ke bina hum "ek saath saare dots" ke baare mein baat nahi kar sakte.

2. Prediction $\hat{y}$ aur line $y=wx+b$

Ek seedhi line ke do knobs hote hain:

Do knobs kyun, ek kyun nahi? Sirf $w$ se line origin $(0,0)$ se guzarne par majboor hoti; real data rarely zero se guzarta hai, isliye $b$ line ko upar ya neeche freely slide karne deta hai.

3. Error aur hum usse square kyun karte hain

Figure mein vertical dashed sticks dekho — har ek ek error hai. Hum chahte hain ki saari sticks ki total length chhoti ho. Lekin ek trap hai:

Toh hum jo quantity minimise karte hain woh squared errors ka sum hai: ${\sum }_{i=1}^n(y_i-{\hat{y}}_i{)}^2$

Symbol ${\sum }_{i=1}^n$ (capital Greek sigma) ka matlab hai "dayi taraf ki cheez ko $i=1,2,\dots ,n$ ke liye jod do." Yeh ek compact "sab jod do" hai.

4. Cost function $J$

$\frac{1}{2n}$ kyun?

• $\frac{1}{n}$ isko ek mean (average) banata hai, taaki score sirf isliye blow up na ho kyunki tumhare paas zyada data hai.
• Extra $\frac{1}{2}$ ek convenience hai: jab hum differentiate karte hain, square upar se $2$ ka factor laata hai, aur $\frac{1}{2}$ usse cancel kar deta hai toh formulas clean rehte hain. Yeh kabhi nahi badalta kahan minimum hai.

5. Vectors aur matrices — bulk notation

Jab bahut saare inputs aur bahut saare points hote hain, toh sums haath se likhna unbearable ho jaata hai. Hum numbers ko grids mein pack karte hain.

• ${\mathbf{x}}_i$ → point $i$ ke saare inputs ek column mein stack kiye hue.
• $\mathbf{X}\in {\mathbb{R}}^{n\times d}$ → design matrix: $n$ rows (ek per point), $d$ columns (ek per input feature). Symbol ${\mathbb{R}}^{n\times d}$ sirf kehta hai "real numbers ki ek grid jisme $n$ rows aur $d$ columns hain."
• $\mathbf{y}\in {\mathbb{R}}^n$ → true answers ka column.
• $\mathbf{w}$ → weights ka column, har feature ke liye ek.

Transpose ${\mathbf{X}}^T$

Matrix-times-vector $\mathbf{Xw}$

Yeh saari predictions ek object mein pack karta hai. $\mathbf{Xw}$ ki row $i$ exactly $w{x}_i+b$ hai. Toh $\mathbf{Xw}$ = guesses $\hat{\mathbf{y}}$ ka poora column ek symbol mein.

Dot product aur ${\mathbf{w}}^T\mathbf{x}$

Norm $\Vert \cdot \Vert$

Yeh Least Squares Estimation ki language hai aur Matrix Pseudoinverse par tikaa hai.

6. Slope, derivative, gradient — downhill compass

Gradient descent mein minus sign kyun? Gradient uphill (zyada cost ki taraf) point karta hai. Hum kam cost chahte hain, isliye hum opposite direction mein step lete hain: $-\nabla J$. Yahi hai ${\mathbf{w}}_{t+1}={\mathbf{w}}_t-\alpha \nabla J$ ka poora logic.

• $\alpha$ = learning rate: kitna bada step lena hai. Zyada bada → valley overshoot karo aur bounce karo; bahut chhota → rengna. Feature Scaling dekho ki kyun unequal input sizes $\alpha$ ko tiny rehne par majboor karti hain.
• Subscript $t$ = iteration number (${\mathbf{w}}_0,{\mathbf{w}}_1,\dots$), ek time step, koi power nahi.
• $ϵ$ (epsilon) = ek tiny threshold; "jab change $ϵ$ se chhota ho tab ruk jao."

7. Yeh sab kaise fit hote hain

Bowl-shaped $J$ ka exactly ek hi bottom hai kyunki yeh convex hai — dekho Convex Optimization. Yahi guarantee hai ki dono methods ek hi best line tak pahunchte hain.

Aage yahi machinery Regularization (Ridge, Lasso), Kalman Filtering, aur Neural Networks mein badhti hai.

Equipment checklist

Dayi taraf cover karo aur oonchi awaaz mein jawab do. Agar koi ruk jaaye, toh woh section dobara padho.