Visual walkthrough — Multiple regression

4.9.23 · D2 · Maths › Probability Theory & Statistics › Multiple regression

Hum sirf assume karte hain ki aap ek graph padh sakte ho aur numbers multiply kar sakte ho. Baaki sab — vectors, "length", "perpendicular", matrices — hum raaste mein earn karte jaayenge.

Step 1 — Hum dekh kya rahe hain? Points aur ek guess

KYA. Hamare paas observations ki ek table hai. Har row ek cheez hai jo humne measure ki: kuch inputs (unhe $x$'s kaho) aur ek output $y$ jise hum predict karna chahte hain. Neeche, sirf ek input imagine karo taaki hum use flat paper pe draw kar sakein — poori kahani baad mein generalise hoti hai.

EK DOT AT A TIME SE SHURU KYUN. Kisi bhi algebra se pehle, jo cheez hum minimise karte hain woh ek gap hai: ek real data point aur hamare guessing-line ke beech ka vertical distance. Agar aap us gap ko picture nahi kar sakte, toh koi bhi formula kuch matlab nahi rakhega.

PICTURE. Amber dots data hain. Cyan line trend ki ek guess hai. Dashed white segments vertical gaps hain — guess har point pe kitni galat hai.

Step 2 — Gaps ko square kyun karte hain

KYA. Hum ek line ko score karne ke liye saare gaps add karte hain. Lekin pehle har gap ko square karte hain: haara score hai $S=\underset{\text{gap 1, squared}}{\underbrace{e_1^2}}+\underset{\text{gap 2, squared}}{\underbrace{e_2^2}}+\cdots +\underset{\text{gap }n}{\underbrace{e_n^2}}={\sum }_{i=1}^n{e}_i^2.$ Yahan $\sum$ sirf "sab add karo" hai, aur $n$ hamare paas kitne data points hain.

SQUARE KYUN, SEEDHA ADD KYUN NAHI? Raw gaps mein do problems hain. (1) $+4$ upar wala point aur $4$ neeche wala point $0$ pe cancel ho jaate — ek terrible line "perfect" score karti. Squaring sign ko khatam kar deti hai kyunki ek square kabhi negative nahi hota. (2) Squaring score ko ek smooth bowl mein bend kar deti hai, taaki calculus uska bottom exactly dhundh sake (Step 6). Absolute value $|e_i|$ bhi signs khatam karta hai lekin ek sharp corner hai — koi clean formula nahi.

PICTURE. Har dashed gap literally ek square ban jaata hai (amber). Total shaded area hi score $S$ hai. Haara poora kaam: line ko slide aur tilt karo jab tak total amber area as small as possible na ho jaaye.

Step 3 — Data ko arrows (vectors) mein stack karna

KYA. $n$ alag-alag numbers ki jagah, saare $y_i$ ko ek tall list mein daalo aur us list ko ek high-dimensional space mein ek single arrow ki tarah treat karo: $\mathbf{y}=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_n\end{array}\right].$ Bold $\mathbf{y}$ ab $n$-dimensional space mein rehne wala ek point hai (har data point ke liye ek axis).

YEH BIZARRE CHEEZ KYUN KARTE HAIN? Kyunki "sum of squared gaps" secretly ek squared length hai. Agar do arrows $\mathbf{u},\mathbf{v}$ ka gap $\mathbf{u}-\mathbf{v}$ hai, toh us gap arrow ki squared length exactly $\sum (u_i-v_i{)}^2$ hai — haare score jaisi hi shape. Toh $S$ minimise karna ban jaata hai: $\mathbf{y}$ ke sabse kareeb wala arrow dhundho. Distance ek aisi cheez hai jiske liye hamare paas strong geometric intuition hai.

PICTURE. Left: teen numbers $y_1,y_2,y_3$. Right: wahi teen numbers 3-D space mein ek arrow $\mathbf{y}$ ke coordinates ki tarah padhe. Wahi data, nayi nazar.

Iske saath, haara score naye roop mein aata hai: $S=\Vert \underset{\text{data arrow}}{\underbrace{\mathbf{y}}}-\underset{\text{prediction arrow}}{\underbrace{\hat{\mathbf{y}}}}{\Vert }^2.$

Step 4 — Haara model kaun se arrows tak pahunch sakta hai? Column space

KYA. Haari prediction hai $\hat{\mathbf{y}}=\mathbf{X}\mathbf{\beta }$. Yahan $\mathbf{X}$ inputs ki table hai (intercept ke liye aage ek column of 1's ke saath), aur $\mathbf{\beta }$ coefficients ki list hai jise hum choose kar sakte hain. $\mathbf{X}\mathbf{\beta }$ multiply karne ka matlab hai: $\mathbf{X}\mathbf{\beta }={\beta }_0\underset{\text{col of 1’s}}{\underbrace{\left[\begin{array}{c}1\\ 1\\ ⋮\\ 1\end{array}\right]}}+{\beta }_1\underset{\text{predictor }1}{\underbrace{\left[\begin{array}{c}{x}_{11}\\ x_{21}\\ ⋮\\ x_{n1}\end{array}\right]}}+\cdots$ Yeh $\mathbf{X}$ ke columns ka ek weighted blend hai, jisme $\beta$'s mixing weights hain.

YEH KYUN MATTER KARTA HAI. Jaise hum $\beta$'s ko saari possible values pe dial karte hain, $\hat{\mathbf{y}}$ un columns ke har blend ko sweep karta hai — aur kuch nahi. Reachable arrows ka woh set ek flat sheet hai (ek plane, generally ek hyperplane) jo origin se guzarta hai. Haara data arrow $\mathbf{y}$ almost kabhi is sheet pe nahi hota (real data noisy hoti hai). Toh hum $\mathbf{y}$ tak pahunch nahi sakte — hum sirf utna hi kareeb ho sakte hain jitna sheet allow kare.

PICTURE. Do input columns ek tilted cyan plane (column space) span karte hain. Data arrow $\mathbf{y}$ (amber) plane se bahar niklta hai. Haari har prediction plane ke upar rehti hai.

Step 5 — Sabse kareeb wala point perpendicular ka foot hai

KYA. Flat sheet pe, $\mathbf{y}$ ke sabse kareeb kaun sa point hai? $\mathbf{y}$ ko seedha sheet pe girate hain. Landing spot $\hat{\mathbf{y}}$ — perpendicular ka foot — sabse kareeb reachable point hai.

PERPENDICULAR KYUN, TILTED KYUN NAHI? Landing point ko sheet ke saath-saath kisi bhi direction mein thoda sa slide karo. Agar connecting arrow $\mathbf{y}-\hat{\mathbf{y}}$ tilted hota (perpendicular nahi), toh uski tilt ki taraf move karne se gap chhota ho jaata — toh aap abhi minimum pe nahi the. Sirf jab gap arrow sheet ke exactly perpendicular khada ho tab koi sideways nudge kaam nahi kar sakta. Yahi hai "least squares" ka geometric matlab.

PICTURE. Amber residual arrow $\mathbf{y}-\hat{\mathbf{y}}$ plane se clean right angle pe milta hai (little white square). Plane pe koi bhi doosra point (grey) $\mathbf{y}$ se door hai — uska dashed connector lamba hai.

Step 6 — "Perpendicular" ko normal equations mein convert karna

KYA. "Residual sheet ke perpendicular hai" matlab hai ki woh $\mathbf{X}$ ke har column ke perpendicular hai (columns sheet ko spread karte hain). Do arrows perpendicular hote hain jab unka dot product zero ho. Un saare "$=0$" statements ko ek saath bundle karo: ${\mathbf{X}}^{\top }(\mathbf{y}-\mathbf{X}\hat{\mathbf{\beta }})=0.$

Term by term:

• ${\mathbf{X}}^{\top }$ — flipped table; uski row $j$ hi $\mathbf{X}$ ka column $j$ hai. Isse multiply karne se har predictor column ka dot product compute hota hai jo bhi baad mein aaye uske saath.
• $\mathbf{y}-\mathbf{X}\hat{\mathbf{\beta }}$ — residual arrow (data minus prediction).
• $0$ — zeros ka column: har predictor residual ke perpendicular hai.

DOT PRODUCT KYUN? Dot product $\mathbf{a}\cdot \mathbf{b}=\sum a_i{b}_i$ overlap measure karta hai; yeh exactly tab zero hota hai jab arrows $90^{\circ }$ pe milte hain. Yeh "residual aur predictors ke beech koi leftover overlap nahi" state karne ka perfect tool hai.

PICTURE. Har predictor column (cyan) residual (amber) ke saath right angle banata hai. Har ek ka residual ke saath dot product zero padhta hai.

Bracket ko multiply karo — ${\mathbf{X}}^{\top }$ ko subtraction ke across distribute karo: $\underset{\text{predictors}\cdot \text{data}}{\underbrace{{\mathbf{X}}^{\top }\mathbf{y}}}-\underset{\text{predictors}\cdot \text{predictors}}{\underbrace{{\mathbf{X}}^{\top }\mathbf{X}}}\hat{\mathbf{\beta }}=0.$

Step 7 — Coefficients ke liye solve karna

KYA. $\hat{\mathbf{\beta }}$ ko isolate karne ke liye, ${\mathbf{X}}^{\top }\mathbf{X}$ se multiply karne ko undo karo. Matrix multiply ko undo karna uske inverse $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ se hota hai (matrix version of "divide"). Ise dono sides pe apply karo: $\hat{\mathbf{\beta }}=\underset{\text{"divide by"}}{\underbrace{({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}}}\underset{\text{right-hand side}}{\underbrace{{\mathbf{X}}^{\top }\mathbf{y}}}.$

INVERSE KYUN, REAL DIVISION KYUN NAHI? Aap matrix se divide nahi kar sakte. Inverse isliye define kiya jaata hai taaki $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}({\mathbf{X}}^{\top }\mathbf{X})=\mathbf{I}$ (identity, "do-nothing" matrix), jo left pe cleanly cancel ho jaata hai aur $\hat{\mathbf{\beta }}$ akela reh jaata hai. Matrix Inverse dekho.

PICTURE. Ek flow: data $\mathbf{y}$ enter karta hai, predictors ke saath dot hota hai (${\mathbf{X}}^{\top }\mathbf{y}$), phir $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ se rescale hota hai, aur bahar aati hai weight list $\hat{\mathbf{\beta }}$ — jo humhe perpendicular ke foot pe land karati hai.

Step 8 — Degenerate case: jab inverse exist nahi karta

KYA. Step 7 ne quietly assume kiya tha ki $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ exist karta hai. Yeh exist karna fail ho jaata hai jab do (ya zyada) predictor columns ek hi direction mein point karte hain — ek doosre ki copy ya scaled sum hai. Yahi perfect collinearity hai.

GEOMETRICALLY KYUN TOOT JAATA HAI. Agar do columns parallel hain, toh woh ek full plane spread out nahi karte; "sheet" ek lower-dimensional line mein collapse ho jaati hai. Ab infinitely many weight-blends ek hi prediction arrow pe land hote hain — aap ek $\beta$ mein add kar sakte ho aur uske twin se subtract kar sakte ho bina koi visible change ke. Koi unique answer nahi, toh koi inverse nahi.

PICTURE. Left (healthy): do columns fan out karte hain aur ek plane pin karte hain; perpendicular ka foot unique hai. Right (collinear): columns ek single line mein overlap kar jaate hain; projection point unpinned hai — woh slide kar sakta hai, toh $\hat{\mathbf{\beta }}$ unique nahi hai.

Ek-picture summary

Upar sab kuch ek single diagram mein collapse ho jaata hai: data arrow $\mathbf{y}$, reachable predictions ki flat sheet ($\mathbf{X}$ ka column space), perpendicular drop jo $\hat{\mathbf{y}}=\mathbf{X}\hat{\mathbf{\beta }}$ deta hai, aur residual $\mathbf{y}-\hat{\mathbf{y}}$ right angle pe khada — jo hai normal equation ${\mathbf{X}}^{\top }(\mathbf{y}-\mathbf{X}\hat{\mathbf{\beta }})=0$, jo hai boxed estimator.

Flashcards