Worked examples — Multiple regression

4.9.23 · D3 · Maths › Probability Theory & Statistics › Multiple regression

Yeh page ek lambi practice hall hai. Hum yahan har tarah ki situation walk through karte hain jo ek multiple-regression problem de sakta hai — clean data, degenerate data, collinear data, ek real-world story, aur ek exam trick — aur har ek ko scratch se solve karte hain. Agar koi symbol yahan aisa hai jo tumne pehle nahi dekha, hum pehle usse build karte hain.

Shuru karne se pehle, parent note se ek reminder: poora game hai $\hat{\mathbf{\beta }}=({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$ solve karna, jahan $\mathbf{X}$ predictors ki table hai (ek leading column of $1$'s ke saath), $\mathbf{y}$ outcome column hai, aur $\hat{\mathbf{\beta }}$ woh numbers ki list hai jo hum chahte hain: intercept aur slopes.

Hume fit judge karne ke liye teen "sum of squares" quantities bhi chahiye hongi. Inhe abhi build karte hain taaki baad mein koi surprise na ho:

---

The scenario matrix

Neeche har problem inhi cells mein se kisi ek mein aati hai. Point yeh hai ki is page ke baad tumne har cell kam se kam ek baar dekhi hai, toh exam mein kuch bhi sach mein naya nahi hoga.

Cell	Kya special banata hai isse	Example
A. Clean full-rank fit	predictors independent, ${\mathbf{X}}^{\top }\mathbf{X}$ invertible	Ex 1
B. Single predictor sanity	multiple-regression machinery ko simple regression tak reduce karna hoga	Ex 2
C. Negative slope	ek predictor jo $y$ ko neeche laata hai	Ex 3
D. Prediction + interpretation	fitted plane mein ek naya point plug karo, partial slope padho	Ex 4
E. Degenerate: perfect collinearity	${\mathbf{X}}^{\top }\mathbf{X}$ singular, inverse exist nahi karta	Ex 5
F. Near-collinearity (VIF blow-up)	inverse exist karta hai but coefficients explode ho jaate hain	Ex 6
G. Fit quality: $R^2$ and adjusted $R^2$	"useless extra predictor" limiting case bhi include karte hue	Ex 7
H. Real-world word problem	ek story ko $\mathbf{X},\mathbf{y}$ mein translate karo	Ex 8
I. Exam twist	centring / rescaling coefficients change karta hai but fit nahi	Ex 9

---

Ex 1 — Cell A: ek clean full-rank fit

Forecast: guess karo — kya slope positive hai, aur kya yeh $1$ se bada hai ya chhota? (Dots generally upar jaate hain.)

1. $\mathbf{X}$ build karo. $1$'s ka leading column, phir $x$ column: $\mathbf{X}=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 3\\ 1& 4\end{array}\right],\mathbf{y}=\left[\begin{array}{c}2\\ 3\\ 5\\ 4\end{array}\right].$ Yeh step kyun? $1$-column woh tarika hai jisse intercept algebra mein enter karta hai — yeh har row mein ${\beta }_0$ se multiply karta hai.

2. ${\mathbf{X}}^{\top }\mathbf{X}$ aur ${\mathbf{X}}^{\top }\mathbf{y}$ banao — aur dekho ki har entry kahan se aati hai. ${\mathbf{X}}^{\top }\mathbf{X}=\left[\begin{array}{cc}n& \sum x_i\\ \sum x_i& \sum x_i^2\end{array}\right]=\left[\begin{array}{cc}4& 10\\ 10& 30\end{array}\right].$ Yahan $n=4$ (chaar rows), $\sum x_i=1+2+3+4=10$, aur $\sum x_i^2=1+4+9+16=30$. ${\mathbf{X}}^{\top }\mathbf{y}=\left[\begin{array}{c}\sum y_i\\ \sum x_i{y}_i\end{array}\right]=\left[\begin{array}{c}2+3+5+4\\ (1)(2)+(2)(3)+(3)(5)+(4)(4)\end{array}\right]=\left[\begin{array}{c}14\\ 39\end{array}\right].$ Toh $14=\sum y_i$ aur $\sum x_i{y}_i=2+6+15+16=39$. Yeh step kyun? ${\mathbf{X}}^{\top }$ (ek matrix jiska pehla row saare $1$'s hai aur doosra row $x$'s hai) ko ek column se multiply karna literally cheezein add karta hai: $1$'s ki row times column ek simple sum hai; $x$-row times column har entry ko $x_i$ se weight karta hai. Yeh exactly wahi sums hain jo simple regression use karta hai.

3. $2\times 2$ system solve karo. Ek $2\times 2$ matrix $\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]$ ke liye inverse hai $\frac{1}{ad-bc}\left[\begin{array}{cc}d& -b\\ -c& a\end{array}\right]$. Yahan $ad-bc=4\cdot 30-10\cdot 10=120-100=20$. $\hat{\mathbf{\beta }}=\frac{1}{20}\left[\begin{array}{cc}30& -10\\ -10& 4\end{array}\right]\left[\begin{array}{c}14\\ 39\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}30(14)-10(39)\\ -10(14)+4(39)\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}420-390\\ -140+156\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}30\\ 16\end{array}\right]=\left[\begin{array}{c}1.5\\ 0.8\end{array}\right].$ Toh ${\hat{\beta }}_0=1.5,{\hat{\beta }}_1=0.8$.

Determinant flag kyun karo? $ad-bc=20\ne 0$, toh inverse genuinely exist karta hai (Matrix Inverse) — machinery sound hai. Safe habit yeh hai ki hamesha ek plug-back check se bachna chahiye (Ex 2).

Verify: VERIFY block exact system symbolically solve karta hai: correct line hai $\hat{y}=1.5+0.8x$, jo $\hat{y}=(2.3,3.1,3.9,4.7)$ deta hai jisme residuals ka sum $0$ hai.

Neeche ki figure yeh chaar points plot karti hai, fit line $1.5+0.8x$ ko yellow mein draw karti hai, aur har residual (dot se line tak pink vertical stick) mark karti hai. Pink sticks dekho: yeh vertical gaps hain jise least squares jitna ho sake chhota karta hai, aur inki signed lengths ka sum zero hota hai.

---

Ex 2 — Cell B: machine ko simple regression tak reduce karna hoga

Forecast: honest slope 1 se kam hona chahiye (dots upar jaate hain but perfect $45^{\circ }$ line par nahi). Guess: around $0.8$?

1. Data centre karo. $\bar{x}=2.5,\bar{y}=3.5$. Deviations $x_i-\bar{x}=(-1.5,-0.5,0.5,1.5)$, $y_i-\bar{y}=(-1.5,-0.5,1.5,0.5)$. Yeh step kyun? Centring slope calculation se intercept hata deta hai — "tilt" ko isolate karta hai. Simple-regression slope formula $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$ ka single-predictor special case hai, toh agar dono disagree karein toh hum se arithmetic slip hui hai.

2. Numerator (covariation) $=(-1.5)(-1.5)+(-0.5)(-0.5)+(0.5)(1.5)+(1.5)(0.5)=2.25+0.25+0.75+0.75=4$.

3. Denominator ($x$ ka spread) $=1.5^2+0.5^2+0.5^2+1.5^2=2.25+0.25+0.25+2.25=5$.

4. Slope ${\hat{\beta }}_1=4/5=0.8$, aur intercept ${\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}=3.5-0.8(2.5)=3.5-2=1.5$.

Yeh Ex 1 se kyun match karta hai: do independent routes — Ex 1 mein matrix inverse aur yahan centred slope formula — dono ${\hat{\beta }}_0=1.5,{\hat{\beta }}_1=0.8$ par land karte hain. Yeh agreement sabse strong sanity check hai jo hamare paas hai.

Verify: $\hat{y}=1.5+0.8x$ ke saath, predictions hain $(2.3,3.1,3.9,4.7)$, residuals $(-0.3,-0.1,1.1,-0.7)$ jo sum to $0$ hote hain. ✓

---

Ex 3 — Cell C: genuinely negative slope

Forecast: zyada maintenance → kam defects, toh slope negative hona chahiye. Uska size guess karo.

1. Sums. $n=4,\sum x=6,\sum x^2=14,\sum y=24,\sum xy=0\cdot 9+1\cdot 7+2\cdot 5+3\cdot 3=26$. Yeh step kyun? Yeh chaar sums hi sab kuch hain jo ${\mathbf{X}}^{\top }\mathbf{X}$ aur ${\mathbf{X}}^{\top }\mathbf{y}$ ko chahiye — exactly wahi pattern $[n,\sum x;\sum x,\sum x^2]$ aur $[\sum y;\sum xy]$ jo humne Ex 1 Step 2 mein build kiya tha.

2. Normal equations. $\left[\begin{array}{cc}4& 6\\ 6& 14\end{array}\right]\hat{\mathbf{\beta }}=\left[\begin{array}{c}24\\ 26\end{array}\right].$

3. Determinant $=4\cdot 14-6\cdot 6=56-36=20\ne 0$, toh $\hat{\mathbf{\beta }}=\frac{1}{20}\left[\begin{array}{cc}14& -6\\ -6& 4\end{array}\right]\left[\begin{array}{c}24\\ 26\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}14(24)-6(26)\\ -6(24)+4(26)\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}336-156\\ -144+104\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}180\\ -40\end{array}\right]=\left[\begin{array}{c}9\\ -2\end{array}\right].$ Toh $\hat{y}=9-2x$.

Negative kyun theek kaam karta hai: least squares kabhi positive slopes demand nahi karta; sign bas data ki tilt follow karta hai.

Verify: $\hat{y}=(9,7,5,3)$ exactly $y$ se match karta hai — points perfectly collinear hain, toh saare residuals $0$ hain aur $R^2=1$. Units: defects per (maintenance hour) $=-2$, meaning har extra hour par 2 kam defects. ✓

---

Ex 4 — Cell D: do predictors, predict aur interpret karo

Forecast: plug in karna ek weighted sum hai; sleep ka effect per hour bas ${\hat{\beta }}_2$ hai.

1. (a) Substitute karo. $\hat{y}=5+8(3)+3(7)=5+24+21=50$. Yeh step kyun? Model linear hai, toh prediction literally coefficient vector aur $(1,x_1,x_2)$ ka dot product hai.

2. (b) Partial slope. $x_2$ ko $+1$ se change karo, $x_1$ fixed rakho: $\Delta \hat{y}={\hat{\beta }}_2\cdot 1=3$. "$x_1$ fixed rakho" kyun: ${\hat{\beta }}_2$ partial slope hai — sleep ka unique contribution study account karne ke baad (Orthogonal Projection ki wajah se yeh "baad" clean hai).

Verify: (a) ko ek chhoti nudge ke saath recompute karo: $x_1=4,x_2=7$ par, $\hat{y}=5+32+21=58$, jo $50+8$ hai — ${\hat{\beta }}_1=8$ per study hour ke saath consistent. Units check: points, aur points-per-hour. ✓

---

Ex 5 — Cell E: perfect collinearity, inverse mar jaata hai

Forecast: do columns same information carry kar rahe hain — machine ko choke karna chahiye.

1. Columns dekho. $x_2$ column exactly $100\times$ $x_1$ column hai. Ek doosre ki copy hai (scaled). Yeh kyun matter karta hai: $\mathbf{X}$ ka column space $x_1$–$x_2$ plane mein "flat" hai — predictors genuine 2D direction span nahi karte.

2. Determinant explicitly compute karo. Sirf do length columns ke saath ek tiny two-row example lo (intercept drop karo clean $2\times 2$ ke liye algebra visible banane ke liye): $x_1=(1,2),x_2=(100,200)$, $\mathbf{X}=\left[\begin{array}{cc}1& 100\\ 2& 200\end{array}\right],{\mathbf{X}}^{\top }\mathbf{X}=\left[\begin{array}{cc}{1}^2+2^2& 1\cdot 100+2\cdot 200\\ 100\cdot 1+200\cdot 2& 100^2+200^2\end{array}\right]=\left[\begin{array}{cc}5& 500\\ 500& 50000\end{array}\right].$ Ab determinant, entry by entry: $\det =(5)(50000)-(500)(500)=250000-250000=0.$ Toh $ad-bc=0$ exactly — do products cancel karte hain kyunki doosra column $100\times$ pehla hai, jo $b=100a$ aur $d=100c$ patterns force karta hai jo $ad$ aur $bc$ ko equal banaate hain.

3. Koi inverse nahi. Kyunki determinant $0$ hai, formula $\frac{1}{ad-bc}[\dots ]$ zero se divide karta hai: $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ exist nahi karta. Infinitely many $\hat{\mathbf{\beta }}$ equally $S$ minimise karte hain — plane redundant direction mein zero cost par freely tilt ho sakta hai. Kyun: ek "flat" matrix se divide karna zero se divide karne jaisa hai — dekho Multicollinearity & VIF aur Positive Semidefinite Matrices (${\mathbf{X}}^{\top }\mathbf{X}$ yahan sirf semidefinite hai, positive definite nahi).

Verify: VERIFY block puri teen-column version ke liye bhi $\det ({\mathbf{X}}^{\top }\mathbf{X})=0$ confirm karta hai. Fix: do length columns mein se ek drop karo; bacha hua fit unique hai. ✓

Neeche ki figure do predictor columns ko arrows ki tarah origin se draw karti hai. Kyunki $x_2=100x_1$, pink arrow blue wale ki direction ke exactly upar lie karta hai (sirf uski length differ karti hai). Ek hi direction mein point karne wale do arrows genuinely 2-D area ka patch frame nahi kar sakte — aur zero area hai ek zero determinant.

---

Ex 6 — Cell F: near-collinearity, coefficients explode ho jaate hain

Forecast: inverse exist karta hai (determinant tiny but nonzero) phir bhi answer unstable hai.

1. Kyun unstable: $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ ek near-zero determinant se divide karta hai, tiny data changes ko huge coefficient swings mein magnify karta hai. Yeh step kyun? Yeh Multicollinearity & VIF ka practical chehra hai.

2. Concrete blow-up. ${\hat{\beta }}_1,{\hat{\beta }}_2$ ke liye least-squares fit ek baar solve karo, phir last outcome $y_4=10$ se $y_4=11$ change karo aur dobara solve karo. Dono slopes ek mere $1$-unit data nudge se substantially shift ho jaate hain — exact rational before/after values, aur swing ka size, VERIFY block mein compute kiye gaye hain. Yeh kyun hota hai: $x_1$ aur $x_2$ sirf last entry mein differ karte hain, toh fit ko us ek distinguishing coordinate par heavily lean karna padta hai; wahan ek tiny wobble dono slopes ko whipsaw kar deta hai.

3. VIF summary. $x_2$ ko $x_1$ par regress karo; near-perfect fit $R_{x_2\mid x_1}^2$ $1$ ke close deta hai, toh variance inflation factor $\text{VIF}=\frac{1}{1-R_{x_2\mid x_1}^2}$ bahut bada hai. $R_{x_2\mid x_1}^2=0.9999$ par yeh $1/0.0001=10000$ ke barabar hai; standard errors $\sqrt{\text{VIF}}\approx 100$ ke saath scale karte hain. High VIF Step 2 ke whipsaw ke liye early-warning gauge hai.

Verify: VERIFY confirm karta hai (i) $\det ({\mathbf{X}}^{\top }\mathbf{X})$ yahan small-but-positive hai (inverse exist karta hai, Ex 5 ki tarah nahi); (ii) do coefficient vectors $1$-unit nudge ke baad noticeably differ karte hain; aur (iii) $1/(1-0.9999)=10000$. ✓

---

Ex 7 — Cell G: $R^2$, adjusted $R^2$, aur useless predictor limit

Forecast: $R^2$ hamesha rise karta hai jab tum variable add karte ho; adjusted $R^2$ junk ke liye fall karna chahiye.

1. (a) $R^2$. $R^2=1-SSE/SST=1-20/100=0.8$. Yeh step kyun? $R^2$ fraction of variance explained hai: leftover error $SSE$ total wobble $SST$ ke share ke roop mein, $1$ se subtract karke.

2. Adjusted $R^2$ with $n=10,p=2$: $R_{adj}^2=1-(1-R^2)\frac{n-1}{n-p-1}=1-0.2\cdot \frac{9}{7}=1-0.2571\dots =\mathrm{0.7429.}$ Yeh step kyun? Degrees of freedom $n-p-1=7$ se divide karna har extra predictor ko penalise karta hai.

3. (b) Junk add karo. Ab $p=3$, $SSE=19.9$, toh $R^2=1-19.9/100=0.801$ (thoda rise hua, hamesha ki tarah). But $R_{adj}^2=1-\frac{19.9/(10-3-1)}{100/(10-1)}=1-\frac{19.9/6}{100/9}=1-\frac{3.3167}{11.111}=1-0.2985=\mathrm{0.7015.}$ Adjusted $R^2$ fall hua $0.7429$ se $0.7015$ tak.

Kyun: useless predictor ne ek degree of freedom cost ki ($n-p-1$ $7\to 6$ drop hua) but error barely reduce ki — ek bura trade.

Verify: VERIFY charon numbers ($0.8,0.7429,0.801,0.7015$) recompute karta hai aur confirm karta hai ki adjusted value drop hua. ✓

---

Ex 8 — Cell H: ek real-world word problem

Forecast: garme din zyada bikta hai (positive ${\beta }_1$), weekends zyada bikta hai (positive ${\beta }_2$).

1. Teen equations set up karo. $\{\begin{array}{l}{\beta }_0+20{\beta }_1+0{\beta }_2=100\\ {\beta }_0+30{\beta }_1+0{\beta }_2=140\\ {\beta }_0+20{\beta }_1+1{\beta }_2=160\end{array}$ Yeh step kyun? Exactly utne hi points jitne unknowns hain aur independent rows ke saath, plane teeno se guzarta hai — koi least-squares averaging ki zaroorat nahi.

2. Eqn 1 eqn 2 se subtract karo: $10{\beta }_1=40\Rightarrow {\beta }_1=4$ ($4 revenue per °C).

3. Eqn 1 eqn 3 se subtract karo: ${\beta }_2=60$ (weekend par $60 extra).

4. Eqn 1 mein back-substitute karo: ${\beta }_0+80=100\Rightarrow {\beta }_0=20$.

Verify: $(30,0)$ plug karo: $20+4(30)+0=140$ ✓; $(20,1)$ plug karo: $20+80+60=160$ ✓. Units: dollars, dollars-per-degree, dollars-per-weekend. ${\beta }_2=60$ ka interpretation: weekend bump temperature fixed rakhte hue — ek clean partial effect. ✓

---

Ex 9 — Cell I: exam twist (centring coefficients change karta hai, fit nahi)

Forecast: slope $-2$ rehna chahiye (steepness axis shift se unaffected hoti hai); intercept change hona chahiye taaki line same points se guzarti rahe.

1. Slope $x$ shifting ke liye invariant hai. $x$ ko $x-c$ se replace karna $\sum (x_i-\bar{x}{)}^2$ aur covariation ko unchanged chhod deta hai, toh ${\hat{\beta }}_1^{\prime }=-2$. Yeh step kyun? Centring vertical axis ko left/right slide karta hai; yeh line ko tilt nahi karta, aur slope formula sirf mean se deviations par depend karta hai, jise shifting alter nahi kar sakti.

2. Naya intercept = $x^{\prime }=0$ par fitted value, yani $x=\bar{x}=1.5$ par: $\hat{y}=9-2(1.5)=6$. Toh ${\hat{\beta }}_0^{\prime }=6$. Kyun: jab $x^{\prime }=0$ hota hai hum $x$ ke mean par baithte hain; ek least-squares line hamesha point $(\bar{x},\bar{y})$ se guzarti hai, aur $\bar{y}=(9+7+5+3)/4=6$. Isliye centred intercept $\bar{y}$ ke barabar hota hai.

3. Fit unchanged. Predictions $\hat{y}=6-2x^{\prime }=6-2(x-1.5)=6-2x+3=9-2x$ — algebraically Ex 3 se identical. Toh $R^2$, har residual, aur har prediction exactly same hai; sirf labels ${\hat{\beta }}_0^{\prime }$ vs ${\hat{\beta }}_0$ moved hue.

Verify: VERIFY confirm karta hai ${\hat{\beta }}_0^{\prime }=6,{\hat{\beta }}_1^{\prime }=-2$ aur ki centred predictions original $(9,7,5,3)$ ke barabar hain. Lesson: predictors ko rescale/centre karna coefficients reshuffle karta hai but fit ki quality kabhi nahi — exam questions se savdhan raho jo ise ek scary-looking coefficient change ke peeche chhupa dete hain.

---

Recall Tumhare liye kaun sa cell sabse hard tha?

Perfect collinearity (Ex 5) fail hoti hai ::: kyunki ${\mathbf{X}}^{\top }\mathbf{X}$ singular hai ($\det =0$), toh $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ exist nahi karta aur $\hat{\mathbf{\beta }}$ unique nahi hai. Adjusted $R^2$ useless predictor ke liye fall karta hai ::: kyunki $n-p-1$ shrink hota hai (lost degree of freedom) jabki $SSE$ barely drop hota hai — ek bura trade. $x$ ko centre karna intercept change karta hai ::: $\bar{y}$ tak (fitted value $x$ ke mean par), jabki slope aur fit unchanged rehti hain.