Exercises — Multiple regression

(a) Intercept constant term hai: ${\hat{\beta }}_0=12$ (predicted $y$ jab dono predictors $0$ hon). (b) $x_1$ ka partial slope uska multiplier hai: ${\hat{\beta }}_1=-4$. Matlab $x_1$ mein 1-unit badhne par, $x_2$ fixed rakhte hue, $y$ 4 se girti hai. (c) Values daalo — model ek weighted sum hai: $\hat{y}=12-4(2)+0.5(10)=12-8+5=9.$

• $\mathbf{X}$ mein har observation ke liye ek row hoti hai aur $p+1=3$ columns hote hain (1-column + $x_1$ + $x_2$): $4\times 3$.
• ${\mathbf{X}}^{\top }\mathbf{X}$ hai $(3\times 4)(4\times 3)=3\times 3$ (square, ek row/col har coefficient ke liye).
• ${\mathbf{X}}^{\top }\mathbf{y}$ hai $(3\times 4)(4\times 1)=3\times 1$ (ek column, $\hat{\mathbf{\beta }}$ ke saath match karta hua).

Square kyun? Square hona inverse ke baare mein baat karne ki zaroori condition hai — lekin kaafi nahi. ${\mathbf{X}}^{\top }\mathbf{X}$ invertible tab hoga jab uska determinant nonzero ho, jo exactly tab hota hai jab $\mathbf{X}$ ki full column rank ho (linearly independent columns, Ex 5.1 mein prove kiya). Square lekin rank-deficient matrix ka $\det =0$ hota hai aur uska koi inverse nahi (Matrix Inverse).

Step 1 — $\mathbf{X}$ banao (pehle 1-column, phir $x$ column): $\mathbf{X}=\left[\begin{array}{cc}1& 0\\ 1& 1\\ 1& 2\end{array}\right],\mathbf{y}=\left[\begin{array}{c}1\\ 3\\ 4\end{array}\right].$ Step 2 — ${\mathbf{X}}^{\top }\mathbf{X}$ form karo. Ones ki row $\cdot$ khud $=n=3$; ones $\cdot$ $x=\sum x=3$; $x\cdot x=\sum x^2=0+1+4=5$: ${\mathbf{X}}^{\top }\mathbf{X}=\left[\begin{array}{cc}3& 3\\ 3& 5\end{array}\right].$ Step 3 — ${\mathbf{X}}^{\top }\mathbf{y}$ form karo. $\sum y=8$; $\sum x_i{y}_i=0+3+8=11$: ${\mathbf{X}}^{\top }\mathbf{y}=\left[\begin{array}{c}8\\ 11\end{array}\right].$ Step 4 — $2\times 2$ ko invert karo. Yeh formula kyun kaam karta hai: ek $2\times 2$ matrix $\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]$ ka inverse hota hai $\frac{1}{ad-bc}\left[\begin{array}{cc}d& -b\\ -c& a\end{array}\right]$. Scalar $ad-bc=\det$ woh (signed) area measure karta hai jo matrix ke columns span karte hain; isse divide karna scaling ko undo karta hai. Rearranged matrix (diagonal swap karo, off-diagonal negate karo) adjugate hai — tum check kar sakte ho ki $\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]\left[\begin{array}{cc}d& -b\\ -c& a\end{array}\right]=(ad-bc)I$ direct multiplication se, isi liye $\det$ se divide karne par $I$ milta hai. Yahan $\det =3\cdot 5-3\cdot 3=6\ne 0$ (to inverse exist karta hai): $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}=\frac{1}{6}\left[\begin{array}{cc}5& -3\\ -3& 3\end{array}\right].$ Step 5 — solve karo: $\hat{\mathbf{\beta }}=\frac{1}{6}\left[\begin{array}{cc}5& -3\\ -3& 3\end{array}\right]\left[\begin{array}{c}8\\ 11\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}40-33\\ -24+33\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}7\\ 9\end{array}\right]=\left[\begin{array}{c}7/6\\ 3/2\end{array}\right].$ To ${\hat{\beta }}_0=\frac{7}{6}\approx 1.167$ aur ${\hat{\beta }}_1=\frac{3}{2}=1.5$. (Yeh exactly Ordinary Least Squares hai ek predictor par — same machinery, matrix-shaped.)

Ek hour $=60$ minutes, to minutes ki value hours ki value ki $60\times$ hai: $x_1^{\text{min}}=60x_1$. Har prediction identical rakhne ke liye, coefficient ko same factor se shrink karna hoga: $8x_1=8\left(\frac{x_1^{\text{min}}}{60}\right)=\frac{8}{60}{x}_1^{\text{min}}\approx 0.1333x_1^{\text{min}}.$ Naya coefficient $=\frac{8}{60}=\frac{2}{15}\approx 0.133$. Fit unchanged raha — sirf woh number badla jo same slope ko relabel karta hai.

Original ($p=3$): $R^2=1-\frac{SSE}{SST}=1-\frac{100}{400}=\mathrm{0.75.}$ $R_{adj}^2=1-(1-R^2)\frac{n-1}{n-p-1}=1-0.25\cdot \frac{19}{16}=1-0.296875=\mathrm{0.703125.}$ Junk predictor add karne ke baad ($p=4$, $SSE=99$): $R_{\text{new}}^2=1-\frac{99}{400}=0.7525(\text{yeh }\text{badh gaya}\text{, jaise hamesha hota hai}).$ $R_{adj,\text{new}}^2=1-(1-0.7525)\frac{19}{15}=1-0.2475\cdot 1.26\overline{6}\approx \mathrm{0.6865.}$ Kya hua: plain $R^2$ badh gaya ($0.75\to 0.7525$) even though predictor kaam ka nahi tha — R² kabhi nahi gir sakta jab tum ek column add karo. Lekin adjusted $R^2$ gir gaya ($0.7031\to 0.6865$) kyunki denominator $n-p-1$ $16$ se $15$ ho gaya, jo waste kiye gaye degree of freedom ko penalise karta hai. Adjusted $R^2$ ne sahi se junk ko flag kiya.

Jab $x_1$ aur $x_2$ nearly parallel hote hain (figure ka left panel), $\mathbf{X}$ ke columns lagbhag ek line par lie karte hain, to ${\mathbf{X}}^{\top }\mathbf{X}$ near-singular ho jaata hai: uska determinant $0$ ke karib hota hai. Inverse compute hota hai $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}=\frac{1}{\det ({\mathbf{X}}^{\top }\mathbf{X})}\mathrm{adj}({\mathbf{X}}^{\top }\mathbf{X})$ ke roop mein, jahan $\mathrm{adj}$ adjugate hai (cofactors ki matrix, transposed). Cofactor entries ordinary size ki rehti hain, lekin sab divided hoti hain ek determinant se jo $0$ ki taraf ja raha hai — isliye inverse ki entries blow up ho jaati hain. Yeh clean scalar proportionality nahi hai; baat yeh hai ki shared factor $1/\det$ har entry ko magnify karta hai. Isliye data mein tiny changes $\hat{\mathbf{\beta }}$ mein huge swings produce karti hain. Geometrically (right panel), fitted plane ko $x_1$ ke along steeply "up" aur $x_2$ ke along equally steeply "down" tilt karne se surface wahan barely move karti hai jahan data hai, to data do slopes ko pin nahi kar sakta: bahut saare wildly different $({\hat{\beta }}_1,{\hat{\beta }}_2)$ lagbhag equally well fit karte hain.

Yeh multicollinearity hai. Isko VIF (variance inflation factor) se quantify kiya jaata hai: predictor $j$ ke liye, ${\text{VIF}}_j=\frac{1}{1-R_j^2},$ jahan $R_j^2$ woh $R^2$ hai jo $x_j$ ko baaki saare predictors par regress karne se milta hai. Agar $x_1$ aur $x_2$ nearly parallel hain, to har ek doosre ko almost perfectly explain karta hai, isliye $R_j^2\to 1$ aur ${\text{VIF}}_j\to \infty$ — woh factor jis se ${\hat{\beta }}_j$ ka variance no-collinearity case ke relative inflate hota hai. Ek common rule of thumb $\text{VIF}>10$ ko flag karta hai.

Orthogonality. Normal equations ${\mathbf{X}}^{\top }\mathbf{X}\hat{\mathbf{\beta }}={\mathbf{X}}^{\top }\mathbf{y}$ se shuru karo. Sab kuch ek side le aao: ${\mathbf{X}}^{\top }\mathbf{y}-{\mathbf{X}}^{\top }\mathbf{X}\hat{\mathbf{\beta }}={\mathbf{X}}^{\top }(\mathbf{y}-\mathbf{X}\hat{\mathbf{\beta }})={\mathbf{X}}^{\top }\mathbf{e}=0.$ ${\mathbf{X}}^{\top }\mathbf{e}=0$ literally kehta hai ki $\mathbf{X}$ ke har column ka $\mathbf{e}$ ke saath dot product zero hai — matlab $\mathbf{e}$ har predictor direction ke perpendicular hai. Yeh orthogonal projection hai: $\hat{\mathbf{y}}=\mathbf{X}\hat{\mathbf{\beta }}$ column space par $\mathbf{y}$ ki shadow hai, $\mathbf{e}$ perpendicular drop hai (Figure s02, red arrow). Residuals zero sum karte hain. Intercept ke saath, $\mathbf{X}$ ka pehla column sab ones ka hai, $1=(1,\dots ,1{)}^{\top }$. ${\mathbf{X}}^{\top }\mathbf{e}=0$ ki uss row mein: $1^{\top }\mathbf{e}={\sum }_{i=1}^n{e}_i=0.$ To positive aur negative vertical gaps exactly cancel ho jaate hain — 1-column include karne ka seedha consequence.

Sab kuch $\bar{y}$ subtract karke center karo. Likho:

• total vector $\mathbf{y}-\bar{y}1$ (length${}^2$ = $SST$),
• explained vector $\hat{\mathbf{y}}-\bar{y}1$ (length${}^2$ = $SSR$),
• residual vector $\mathbf{y}-\hat{\mathbf{y}}=\mathbf{e}$ (length${}^2$ = $SSE$). Yeh add hote hain: $(\mathbf{y}-\bar{y}1)=(\hat{\mathbf{y}}-\bar{y}1)+\mathbf{e}$. Squared length lo: $\Vert \mathbf{y}-\bar{y}1{\Vert }^2=\Vert \hat{\mathbf{y}}-\bar{y}1{\Vert }^2+\Vert \mathbf{e}{\Vert }^2+2(\hat{\mathbf{y}}-\bar{y}1{)}^{\top }\mathbf{e}.$ Cross term vanish ho jaata hai: $\hat{\mathbf{y}}-\bar{y}1$ column space of $\mathbf{X}$ mein rehta hai (dono $\hat{\mathbf{y}}=\mathbf{X}\hat{\mathbf{\beta }}$ aur $\bar{y}1$, $\mathbf{X}$ ke columns ke combinations hain), aur $\mathbf{e}\perp$ us poore space ke Ex 4.1 se. To $2(\cdots {)}^{\top }\mathbf{e}=0$, deta hai: $\underset{SST}{\underbrace{\Vert \mathbf{y}-\bar{y}1{\Vert }^2}}=\underset{SSR}{\underbrace{\Vert \hat{\mathbf{y}}-\bar{y}1{\Vert }^2}}+\underset{SSE}{\underbrace{\Vert \mathbf{e}{\Vert }^2}}.$ Yeh exactly $a^2+b^2=c^2$ hai ek right triangle ke liye jiska right angle $\hat{\mathbf{y}}$ par hai — same right angle Figure s02 se.

PSD. Kisi bhi vector $\mathbf{v}$ ke liye: ${\mathbf{v}}^{\top }({\mathbf{X}}^{\top }\mathbf{X})\mathbf{v}=(\mathbf{Xv}{)}^{\top }(\mathbf{Xv})=\Vert \mathbf{Xv}{\Vert }^2\ge 0.$ Squared length kabhi negative nahi hoti, isliye ${\mathbf{X}}^{\top }\mathbf{X}$ positive semidefinite hai. Hamesha. Positive definite iff full column rank. $\Vert \mathbf{Xv}{\Vert }^2=0$ force karta hai $\mathbf{Xv}=0$.

• Agar columns linearly independent hain, to sirf $\mathbf{v}=0$ hi $\mathbf{Xv}=0$ deta hai. To ${\mathbf{v}}^{\top }{\mathbf{X}}^{\top }\mathbf{Xv}>0$ har $\mathbf{v}\ne 0$ ke liye → positive definite → invertible → unique $\hat{\mathbf{\beta }}$.
• Agar columns dependent hain (perfect collinearity), koi $\mathbf{v}\ne 0$ $\mathbf{Xv}=0$ deta hai, to ${\mathbf{v}}^{\top }{\mathbf{X}}^{\top }\mathbf{Xv}=0$ with $\mathbf{v}\ne 0$ → sirf semidefinite, $\det =0$, invertible nahi. Formula $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ exist nahi karta; infinitely many $\hat{\mathbf{\beta }}$ $S$ minimize karte hain.

Objective hai $S(\mathbf{\beta })=\Vert \mathbf{y}-\mathbf{X}\mathbf{\beta }{\Vert }^2$, ek quadratic jiska Hessian (second derivatives ki matrix) hai $2{\mathbf{X}}^{\top }\mathbf{X}$. Ex 5.1 se yeh positive definite hai jab columns independent hain. Positive-definite Hessian wala quadratic strictly convex hota hai: yeh har direction mein upar curve karta hai, isliye iska exactly ek lowest point hai aur koi doosra stationary point nahi. Woh single point woh hai jahan gradient $-2{\mathbf{X}}^{\top }\mathbf{y}+2{\mathbf{X}}^{\top }\mathbf{X}\mathbf{\beta }$ vanish hota hai — matlab normal equations — unique $\hat{\mathbf{\beta }}=({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$ deta hai. Gauss–Markov Theorem ko "the" estimator ki baat karne ke liye exactly yahi chahiye. Simple Linear Regression se compare karo, jahan same convexity ek best line deti hai.

Jaane se pehle ek-line answers. Ex 2.1 mein ${\hat{\beta }}_0$ aur ${\hat{\beta }}_1$ kya hain? ::: $7/6$ aur $3/2$. Ex 3.1 mein, adjusted $R^2$ kyun girta hai jab plain $R^2$ badhta hai? ::: Degrees-of-freedom penalty $n-p-1$ useless predictor ke liye $SSE$ ke girne se zyada tezi se shrink karti hai. Kaunsi ek equation prove karti hai ki residual $\perp$ $\mathbf{X}$ ke har column ke saath hai? ::: ${\mathbf{X}}^{\top }\mathbf{e}=0$ (normal equations rearranged). ${\mathbf{X}}^{\top }\mathbf{X}$ positive definite (sirf PSD nahi) kab hota hai? ::: Exactly jab $\mathbf{X}$ ke linearly independent columns hon (full column rank).