Worked examples — Linear regression — normal equation, gradient descent derivation

5.6.1 · D3 · Coding › Machine Learning (Aerospace Applications) › Linear regression — normal equation, gradient descent deriva

Yeh deep dive parent topic ka hands-on companion hai. Wahan humne ek line fit karne ke do tarike derive kiye the. Yahan hum un machines ko har tarah ke input par run karte hain jo tum meet kar sakte ho — clean data, perfect fits, broken (singular) data, badly scaled data, real-world word problems, aur ek exam twist.

Yeh page self-contained hai: jo bhi hum use karte hain, woh pehle yahan define kiya gaya hai.

Recall Woh do formulas jinhe hum baar baar use karenge

Closed form (ek shot mein): ${\mathbf{w}}^{\ast }=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$. Iterative: ${\mathbf{w}}_{t+1}={\mathbf{w}}_t-\frac{\alpha }{n}{\mathbf{X}}^T(\mathbf{X}{\mathbf{w}}_t-\mathbf{y})$. Yahan $\alpha$ learning rate hai (step size). $\frac{1}{n}$ isliye aata hai kyunki average cost ke gradient mein wahi $\frac{1}{n}$ hota hai — hum us averaged slope ke against step karte hain.

---

Scenario matrix

Ek line-fitting problem tumhare saamne limited tarah ke flavours mein aa sakti hai. Yeh poora menu hai. Har flavour ko neeche kam se kam ek fully worked example milta hai.

Cell	Case class	Kya special hai	Example
A	Positive slope, imperfect fit	ordinary case, errors cancel nahi karte	Ex 1
B	Perfectly collinear points	error exactly zero hai; fit sabse guzarti hai	Ex 2
C	Negative slope	$w$ ka sign negative hai — sign slips pakadta hai	Ex 3
D	Zero slope (flat)	best line horizontal hai, $w=0$	Ex 4
E	Singular / degenerate ${\mathbf{X}}^T\mathbf{X}$	inverse exist nahi karta (sab $x$ equal hain)	Ex 5
F	Gradient descent, step-by-step	iterative path, learning rate behaviour	Ex 6
G	Learning rate too large (divergence)	$\alpha$ ka limiting behaviour	Ex 7
H	Badly scaled features	kyun Feature Scaling matter karta hai	Ex 8
I	Real-world word problem (aerospace)	units, interpretation	Ex 9
J	Exam twist: proof/geometry	residual ki orthogonality	Ex 10

---

Example 1 — Cell A: positive slope, imperfect fit

Forecast: points left-to-right rise karte hain, toh guess karo $w\approx 1.5$ aur ek chota positive $b$. Aage padhne se pehle apna guess likhlo.

1. $\mathbf{X}$ aur $\mathbf{y}$ banao. $\mathbf{X}=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 3\end{array}\right],\mathbf{y}=\left[\begin{array}{c}2\\ 4\\ 5\end{array}\right]$ Yeh step kyun? $1$s ka pehla column bias $b$ ko $\mathbf{w}=[b,w{]}^T$ ke andar rehne deta hai, taaki ek formula dono knobs handle kare.

2. ${\mathbf{X}}^T\mathbf{X}$ aur ${\mathbf{X}}^T\mathbf{y}$ banao. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}3& 6\\ 6& 14\end{array}\right],{\mathbf{X}}^T\mathbf{y}=\left[\begin{array}{c}11\\ 25\end{array}\right]$ Yeh step kyun? Normal equation ${\mathbf{X}}^T\mathbf{X}\mathbf{w}={\mathbf{X}}^T\mathbf{y}$ ko exactly yahi do objects chahiye. ${\mathbf{X}}^T\mathbf{X}$ saare inputs ko ek $2\times 2$ summary mein collapse karta hai (row·column sums: top-left hai $1+1+1=3$, bottom-right hai $1+4+9=14$); ${\mathbf{X}}^T\mathbf{y}$ har input column ko answers ke saath pair karta hai ($1\cdot 2+1\cdot 4+1\cdot 5=11$, aur $1\cdot 2+2\cdot 4+3\cdot 5=25$).

3. $2\times 2$ ko invert karo. Determinant $=3\cdot 14-6\cdot 6=6$. $({\mathbf{X}}^T\mathbf{X}{)}^{-1}=\frac{1}{6}\left[\begin{array}{cc}14& -6\\ -6& 3\end{array}\right]$ Yeh step kyun? $\mathbf{w}$ ko isolate karne ke liye hume multiply-by-${\mathbf{X}}^T\mathbf{X}$ ko undo karna hai; $2\times 2$ ke liye diagonal swap karo, off-diagonal negate karo, determinant se divide karo.

4. Multiply out karo. ${\mathbf{w}}^{\ast }=\frac{1}{6}\left[\begin{array}{cc}14& -6\\ -6& 3\end{array}\right]\left[\begin{array}{c}11\\ 25\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}4\\ 9\end{array}\right]=\left[\begin{array}{c}0.6\overline{6}\\ 1.5\end{array}\right]$ Toh $b=0.667,w=1.5$.

Verify: ${\hat{y}}_1=1.5(1)+0.667=2.167$, ${\hat{y}}_2=3.667$, ${\hat{y}}_3=5.167$. Residuals $-0.167,+0.333,-0.167$ ka sum $\approx 0$ — exactly yahi ek least-squares fit force karta hai. Slope ne hamare forecast se match kiya. ✓

Figure — aise padhein: black dots teen data points hain, red line fitted $\hat{y}=1.5x+0.667$ hai, aur har dashed vertical segment ek residual hai (ek dot red line ke kitna upar ya neeche hai). Notice karo ki beech wala dot line ke upar hai jabki bahar ke dono neeche hain — yeh visual balance residuals ka zero tak sum hona hai.

---

Example 2 — Cell B: perfectly collinear data

Forecast: agar points already ek line par hain, toh "best" line wahi line hai, aur total error exactly zero hona chahiye.

1. Banao. $\mathbf{X}=\left[\begin{array}{cc}1& 0\\ 1& 1\\ 1& 2\end{array}\right]$, $\mathbf{y}=\left[\begin{array}{c}1\\ 3\\ 5\end{array}\right]$.
2. Summaries. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}3& 3\\ 3& 5\end{array}\right]$, ${\mathbf{X}}^T\mathbf{y}=\left[\begin{array}{c}9\\ 13\end{array}\right]$. Yeh step kyun? Ex 1 jaisi hi normal-equation recipe — top-left $=1+1+1=3$, bottom-right $=0+1+4=5$; "perfect data" procedure ko kuch nahi badalta.
3. Determinant $=3\cdot 5-3\cdot 3=6$, toh $({\mathbf{X}}^T\mathbf{X}{)}^{-1}=\frac{1}{6}\left[\begin{array}{cc}5& -3\\ -3& 3\end{array}\right].$ Yeh step kyun? Ex 1 jaisi hi isolate-$\mathbf{w}$ wajah: multiply undo karne ke liye invert karo.
4. Solve karo. ${\mathbf{w}}^{\ast }=\frac{1}{6}\left[\begin{array}{cc}5& -3\\ -3& 3\end{array}\right]\left[\begin{array}{c}9\\ 13\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}6\\ 12\end{array}\right]=\left[\begin{array}{c}1\\ 2\end{array}\right]$. Toh $b=1,w=2$.

Verify: $\hat{y}=2x+1$ exactly $1,3,5$ deta hai — residual vector $0$ hai, cost $J=0$. Fit exact hai, forecast se match karta hai. ✓

---

Example 3 — Cell C: negative slope

Forecast: downhill data → $w<0$. $x$ mein $+1$ ka har step $y$ ko $2$ se girata hai, toh expect karo $w\approx -2$, $b\approx 7$.

1. Banao. $\mathbf{X}=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 3\end{array}\right]$, $\mathbf{y}=\left[\begin{array}{c}5\\ 3\\ 1\end{array}\right]$.
2. Summaries. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}3& 6\\ 6& 14\end{array}\right]$ (Ex 1 jaisi hi $x$'s!), ${\mathbf{X}}^T\mathbf{y}=\left[\begin{array}{c}9\\ 14\end{array}\right]$. Yeh step kyun? ${\mathbf{X}}^T\mathbf{X}$ sirf inputs par depend karta hai, isliye Ex 1 ka inverse reuse karte hain; sirf ${\mathbf{X}}^T\mathbf{y}$ badalta hai ($1\cdot 5+1\cdot 3+1\cdot 1=9$, $1\cdot 5+2\cdot 3+3\cdot 1=14$).
3. Solve karo. ${\mathbf{w}}^{\ast }=\frac{1}{6}\left[\begin{array}{cc}14& -6\\ -6& 3\end{array}\right]\left[\begin{array}{c}9\\ 14\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}42\\ -12\end{array}\right]=\left[\begin{array}{c}7\\ -2\end{array}\right]$. Toh $b=7,w=-2$.

Verify: $\hat{y}=-2x+7$ → $5,3,1$: exact fit hai, aur $w=-2<0$ negative slope confirm karta hai jo haari aankhon ne predict kiya tha. ✓ Lesson: negative answer koi error nahi hai — sign direction encode karta hai.

---

Example 4 — Cell D: zero slope (flat line)

Forecast: $y$ completely $x$ ko ignore karta hai, toh best line flat hai: $w=0$, $b=4$.

1. Summaries. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}3& 6\\ 6& 14\end{array}\right]$, ${\mathbf{X}}^T\mathbf{y}=\left[\begin{array}{c}12\\ 24\end{array}\right]$. Yeh step kyun? $x$'s Ex 1 jaisi hain toh ${\mathbf{X}}^T\mathbf{X}$ unchanged hai; ${\mathbf{X}}^T\mathbf{y}$ answers ko har column ke against sum karta hai ($4+4+4=12$, $1\cdot 4+2\cdot 4+3\cdot 4=24$). Hum yeh dono isliye banate hain kyunki normal equation ko exactly yahi feed hota hai.
2. Solve karo. ${\mathbf{w}}^{\ast }=\frac{1}{6}\left[\begin{array}{cc}14& -6\\ -6& 3\end{array}\right]\left[\begin{array}{c}12\\ 24\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}24\\ 0\end{array}\right]=\left[\begin{array}{c}4\\ 0\end{array}\right]$. Yeh step kyun? $w$ ke respect mein gradient tabhi zero hota hai jab slope kuch contribute nahi karta — algebra exactly $w=0$ de deta hai.

Verify: $\hat{y}=4$ har jagah, residuals sab $0$. Flat target genuinely flat fit produce karta hai. ✓

---

Example 5 — Cell E: singular / degenerate input

Forecast: saare points vertical line $x=2$ par hain. Ek function $\hat{y}=wx+b$ vertical nahi ho sakta. Machine ke tootne ki expect karo — inverse exist nahi karna chahiye.

1. Banao. $\mathbf{X}=\left[\begin{array}{cc}1& 2\\ 1& 2\\ 1& 2\end{array}\right]$, $\mathbf{y}=\left[\begin{array}{c}3\\ 5\\ 9\end{array}\right]$.
2. Summaries. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}3& 6\\ 6& 12\end{array}\right]$. Yeh step kyun? Same recipe; note karo ki $x$-column exactly $1$-column ka $2\times$ hai, ek warning sign jo agli step mein feel hoga.
3. Determinant. $3\cdot 12-6\cdot 6=36-36=0$. Yeh step kyun? Zero determinant matlab matrix ka koi inverse nahi — normal equation formula $({\mathbf{X}}^T\mathbf{X}{)}^{-1}$ undefined hai. Yeh multicollinearity hai: bias column aur $x$-column same information carry karte hain (ek doosre ka $2\times$ hai), toh hum $b$ aur $w$ alag nahi kar sakte.

Verify: determinant literally $0$ hai, forecast confirm karta hai. Fixes: redundant column drop karo, ya thoda penalty add karo — dekho Regularization (Ridge, Lasso), jo ${\mathbf{X}}^T\mathbf{X}$ ko ${\mathbf{X}}^T\mathbf{X}+\lambda \mathbf{I}$ se replace karta hai taaki determinant kabhi zero na ho. ✓

Figure — aise padhein: red vertical line woh jagah hai jahan teeno black dots rehte hain ($x=2$). $\hat{y}=wx+b$ form ki koi bhi sloped-ya-flat line vertical trace nahi kar sakti — yeh impossibility exactly zero determinant hai ek picture ke roop mein.

---

Example 6 — Cell F: gradient descent, do honest steps

Forecast: zero se start karte hue, pehla gradient bada aur negative hoga (predictions data se kaafi neeche hain), toh $\mathbf{w}$ upar jump karega. Expect karo ki yeh Ex 1 ke true answer $[0.667,1.5]$ ki taraf crawl karega.

1. Iteration 0 — predict karo phir error measure karo. $\hat{\mathbf{y}}=\mathbf{X}{\mathbf{w}}_0=[0,0,0{]}^T$, toh error $\mathbf{e}=\mathbf{X}{\mathbf{w}}_0-\mathbf{y}=[-2,-4,-5{]}^T$. Yeh step kyun? Gradient $\nabla J=\frac{1}{n}{\mathbf{X}}^T\mathbf{e}$ ko pehle error chahiye — konsi direction mein step lena hai yeh decide karne se pehle hume jaanna hai ki hum kitne galat hain.
2. Gradient. $\nabla J=\frac{1}{3}{\mathbf{X}}^T\mathbf{e}=\frac{1}{3}\left[\begin{array}{ccc}1& 1& 1\\ 1& 2& 3\end{array}\right]\left[\begin{array}{c}-2\\ -4\\ -5\end{array}\right]=\frac{1}{3}\left[\begin{array}{c}-11\\ -25\end{array}\right]=\left[\begin{array}{c}-3.667\\ -8.333\end{array}\right].$
3. Update. ${\mathbf{w}}_1={\mathbf{w}}_0-\alpha \nabla J=[0,0]-0.1[-3.667,-8.333]=[0.3667,0.8333]$. Yeh step kyun? Hum gradient ke against step karte hain (cost surface par downhill); $\alpha$ scale karta hai ki kitna dur.
4. Iteration 1. $\hat{\mathbf{y}}=\mathbf{X}{\mathbf{w}}_1=[1.2,2.0333,2.8667{]}^T$, error $\mathbf{e}=[-0.8,-1.9667,-2.1333{]}^T$. Gradient $=\frac{1}{3}{\mathbf{X}}^T\mathbf{e}$: pehla component $\frac{1}{3}(-0.8-1.9667-2.1333)=\frac{1}{3}(-4.9)=-1.6333$; doosra component $\frac{1}{3}(1(-0.8)+2(-1.9667)+3(-2.1333))=\frac{1}{3}(-11.1333)=-3.7111$. Update ${\mathbf{w}}_2={\mathbf{w}}_1-0.1[-1.6333,-3.7111]=[0.5300,1.2044]$. Yeh step kyun? Exactly wohi rule baar baar — descent sirf "error measure karo, downhill step lo" baar baar hai.

Verify: ${\mathbf{w}}_0=[0,0]\to {\mathbf{w}}_1=[0.367,0.833]\to {\mathbf{w}}_2=[0.530,1.204]$. Har step closed-form target $[0.667,1.5]$ ki taraf move karta hai — exactly woh crawl jo humne forecast kiya tha. ✓

Figure — aise padhein: axes do knobs hain ($b$ horizontal, $w$ vertical). Black path $w_0\to w_1\to w_2$ pe hop karta hai, aur red star exact answer hai. Dekho ki kaise har hop shrink hota hai aur star ki taraf bend karta hai — yeh convergence hai.

---

Example 7 — Cell G: learning rate bahut bada

Forecast: ek giant step valley ko overshoot karega aur doosri wall par aur upar land karega. Cost $J$ worse honi chahiye, aur repeated steps blow up karenge.

1. Ek update. ${\mathbf{w}}_1=[0,0]-2[-3.667,-8.333]=[7.333,16.667]$. Yeh step kyun? $\alpha =2$ step ko multiply karta hai; sign abhi bhi correct hai (downhill), lekin magnitude bahut zyada badi hai.
2. Cost check karo. $J(\mathbf{w})=\frac{1}{2n}\Vert \mathbf{Xw}-\mathbf{y}{\Vert }^2$ use karke $n=3$ ke saath: $J_0=\frac{1}{6}(2^2+4^2+5^2)=\frac{45}{6}=7.5$. ${\mathbf{w}}_1$ ke saath: $\hat{\mathbf{y}}=[24,40.667,57.333]$, residuals $\approx [22,36.667,52.333]$, $J_1=\frac{1}{6}(22^2+36.667^2+52.333^2)\approx 762$. Yeh step kyun? $J$ ko before aur after compare karne se pata chalta hai ki step ne help kiya ya nahi. Yahan toh woh explode ho gaya.

Verify: $J_1\approx 762\gg J_0=7.5$ — cost ~100× badh gayi, divergence confirm karti hai. Rule of thumb: agar $J$ increase kare, toh $\alpha$ aadha kar do. Convex Optimization guarantee karta hai ki ek valley exist karta hai; bahut bade steps usse find nahi kar sakte. ✓

---

Example 8 — Cell H: badly scaled features

Forecast: metre-feature ka gradient component fraction ka daba dega, toh ek knob race karega jabki doosra barely move karega — descent zig-zag karega.

1. Gradient component per feature. Ek point ke liye, $\nabla J=(\hat{y}-y)\mathbf{x}=(0-10)(8000,0.5)=(-80000,-5)$. Yeh step kyun? Gradient error times feature value hota hai; ek bada feature value bada component banata hai.
2. Ratio. $\frac{80000}{5}=16000$. Yeh step kyun? Do knobs $16000$ factor se alag forces feel karte hain; koi bhi single $\alpha$ dono mein se ek ke liye galat hoga.

Verify: component ratio $16000$ feature-magnitude ratio $8000/0.5=16000$ se exactly match karta hai — proof ki imbalance purely scale se aata hai. Fix: pehle features standardise karo — dekho Feature Scaling. Scaling ke baad, dono components comparable hote hain aur descent seedha downhill jaata hai. ✓

---

Example 9 — Cell I: aerospace word problem

Forecast: humne yeh data already fit kiya tha (Ex 1): $\hat{y}=1.5x+0.667$. $x=4$ par expect karo $\approx 6.7$ kg/s.

1. Fit reuse karo. $w=1.5\text{(kg/s per V)}$, $b=0.667\text{kg/s}$. Yeh step kyun? Calibration = ek baar fit karo, phir new readings ke liye line ko lookup ki tarah use karo.
2. $x=4$ par predict karo. $\hat{y}=1.5(4)+0.667=6.667\text{kg/s}$. Yeh step kyun? Hum data se ek volt bahar extrapolate kar rahe hain — allowed hai kyunki physical relation yahan linear hai.

Verify: units check — slope kg/s per volt hai, volts se multiply karo toh kg/s milta hai; bias kg/s hai; sum kg/s hai. ✓ Answer $6.667$ kg/s forecast se match karta hai. (Practice mein hum $x=4$ ko $1$–$3$ V calibration range se thoda bahar flag karenge.)

---

Example 10 — Cell J: exam twist (geometry proof)

Forecast: normal equation condition ${\mathbf{X}}^T(\mathbf{y}-\mathbf{Xw})=0$ se hi janma tha, toh yeh exactly hold hona chahiye — geometry hai, luck nahi.

1. Residuals compute karo. Ex 1 se, $\hat{\mathbf{y}}=[2.167,3.667,5.167]$, toh $\mathbf{r}=\mathbf{y}-\hat{\mathbf{y}}=[-0.167,0.333,-0.167]$. Yeh step kyun? Perpendicularity $\mathbf{r}$ ke baare mein ek statement hai, toh pehle $\mathbf{r}$ numbers mein chahiye.
2. Column 1 se dot karo ($1$s wala). $(-0.167)+(0.333)+(-0.167)=0$. Yeh step kyun? All-ones column se dot karna bias direction mein perpendicularity test karta hai — geometrically matlab hai ki residuals ka sum zero hona chahiye (koi leftover vertical offset nahi jo bias abhi bhi absorb kar sake).
3. Column 2 se dot karo ($x$'s wala). $1(-0.167)+2(0.333)+3(-0.167)=-0.167+0.667-0.5=0$. Yeh step kyun? $x$-column se dot karna slope direction mein perpendicularity test karta hai — matlab hai ki errors mein koi leftover linear trend nahi jo slope abhi bhi exploit kar sake. Dono dots zero hone ka matlab hai ${\mathbf{X}}^T\mathbf{r}=0$, proof complete.

Verify: dono dot products $0$ hain, toh ${\mathbf{X}}^T\mathbf{r}=0$. Geometrically, $\mathbf{X}{\mathbf{w}}^{\ast }$ $\mathbf{y}$ ka shadow (orthogonal projection) hai $\mathbf{X}$ ke columns se spanned plane par; bacha hua $\mathbf{r}$ us plane se seedha bahar point karta hai. Yeh Least Squares Estimation ka dil hai. ✓

Figure — aise padhein: black horizontal arrow $\mathbf{X}{\mathbf{w}}^{\ast }$ column space ke andar lie karta hai (black line ke roop mein draw kiya); tall black arrow $\mathbf{y}$ hai; red arrow residual $\mathbf{r}$ hai jo $\mathbf{X}{\mathbf{w}}^{\ast }$ ki tip se $\mathbf{y}$ tak jaata hai. Chhota square dikhata hai ki red arrow column space se right angle par milta hai — woh right angle hi ${\mathbf{X}}^T\mathbf{r}=0$ hai.

---

Recall Self-test

Kis cell mein koi inverse nahi hai aur kyun? ::: Cell E — saab $x$ equal hone se $\det ({\mathbf{X}}^T\mathbf{X})=0$ hota hai (bias aur feature columns proportional hain). Agar gradient descent ki cost $J$ badhe, toh kya karte ho? ::: Learning rate $\alpha$ aadha kar do (Cell G). Negative $w$ ka matlab kya hai? ::: Downhill data — $x$ badhne par $y$ kam hota hai (Cell C). Residuals $\mathbf{X}$ ke columns ke orthogonal kyun hone chahiye? ::: Warna ek leftover trend abhi bhi remove kiya ja sakta tha, toh fit optimal nahi tha (Cell J).