Multiple regression

4.9.23 · Maths › Probability Theory & Statistics

YEH HAI KYA?

"Partial" kyun? Simple regression mein ek slope $x$ ka direct effect aur jo bhi cheez $x$ ke saath correlate karti hai, dono ko mix kar deta hai. Model mein kai predictors hone par, $β_{j}$ sirf $x_{j}$ ka effect measure karta hai jab baaki predictors ne jo kuch explain kar saka, woh kar liya ho — yahi uska unique contribution hai.

Matrix form — kyun hum notation switch karte hain

$n$ equations ek-ek karke likhna bahut mushkil hai. Inhe stack karo.

1's ka column ISSI TARAH intercept ko matrix algebra mein fold karta hai (yeh har row mein $β_{0}$ se multiply hota hai).

DERIVATION scratch se — normal equations

Goal: $β$ ko is tarah choose karo ki sum of squared residuals (vertical gaps) minimum ho: $S (β) = \sum_{i = 1}^{n} ε_{i}^{2} = ∥ y - X β ∥^{2} = (y - X β)^{⊤} (y - X β) .$

Squares kyun? Yeh smooth (differentiable) hote hain, bade misses ko zyada penalise karte hain, aur ek unique closed-form answer dete hain.

Step 1 — expand karo. $S = y^{⊤} y - 2 β^{⊤} X^{⊤} y + β^{⊤} X^{⊤} X β .$ Yeh step kyun? Dono cross terms $y^{⊤} X β$ aur $β^{⊤} X^{⊤} y$ equal scalars hain (ek doosre ke transpose), isliye woh $- 2$ term mein combine ho jaate hain.

Step 2 — vector $β$ ke w.r.t. differentiate karo aur zero set karo. $\frac{\partial}{\partial β} (a^{⊤} β) = a$ aur $\frac{\partial}{\partial β} (β^{⊤} A β) = 2 A β$ (symmetric $A = X^{⊤} X$ ke liye) use karte hue: $\frac{\partial S}{\partial β} = - 2 X^{⊤} y + 2 X^{⊤} X β = 0 .$ Zero kyun set karte hain? Minimum par gradient zero hota hai; $S$ convex hai (ek quadratic with PSD Hessian $2 X^{⊤} X$ ), isliye yeh stationary point global minimum hai.

Step 3 — normal equations. $X^{⊤} X \hat{β} = X^{⊤} y$

Step 4 — solve karo (agar $X^{⊤} X$ invertible hai, yaani predictors perfectly collinear nahi hain): $\hat{β} = (X^{⊤} X)^{- 1} X^{⊤} y$

Fit measure karna: $R^{2}$ aur adjusted $R^{2}$

Problem: koi bhi predictor (chahe random hi ho) add karne par $R^{2}$ kabhi kam nahi hota. Isliye hum complexity ko penalise karte hain:

Worked Example 1 — chhota 2-predictor fit by hand-logic

Data: exam score $y$ predict karo hours studied $x_{1}$ aur hours slept $x_{2}$ se.

$i$	$x_{1}$	$x_{2}$	$y$
1	1	6	50
2	2	7	65
3	3	5	70
4	4	8	85

Hum $X$ banate hain ek leading 1-column ke saath, $X^{⊤} X$ (ek $3 \times 3$ matrix) aur $X^{⊤} y$ form karte hain, phir $\hat{β} = (X^{⊤} X)^{- 1} X^{⊤} y$ solve karte hain.

1-column kyun include karte hain? Iske bina hum surface ko origin se force karte hain — jo usually galat hota hai.
Linear system kyun solve karte hain, eyeball kyun nahi? 2+ predictors ke saath koi simple ratio nahi hota; predictors information share karte hain, isliye saare slopes jointly solve karne padte hain.

(Neeche ka VERIFY block is exact system ko symbolically solve karta hai.)

Worked Example 2 — partial slope interpret karna

Maano ek fit deta hai $\overset{y}{^} = 5 + 8 x_{1} + 3 x_{2}$ jahaan $x_{1}$ =hours studied, $x_{2}$ =hours slept.

$\hat{β}_{1} = 8$ : ek extra ghante ki study 8 points add karti hai, sleep constant rakhte hue. "Holding constant" kyun matter karta hai: agar jo students zyada padhte hain woh zyada sote bhi hain, toh $y$ ka sirf $x_{1}$ par simple regression galti se study ko sleep ka benefit bhi credit kar dega.
$x_{1} = 3, x_{2} = 7$ ke liye predict karo: $\overset{y}{^} = 5 + 24 + 21 = 50$ . Kyun: bas plug in karo; model linear hai isliye yeh ek weighted sum hai.

Common mistakes (Steel-man + fix)

Recall Feynman: ek 12-saal ke bacche ko samjhao

Socho tum guess kar rahe ho ki ek plant kitna lamba badhega yeh dekhke ki use kitna paani aur kitni dhoop mili. Tum kai plants collect karte ho aur in dots mein se best flat "ramp" draw karte ho taaki dots aur ramp ke upar-neeche ki distances smallest hon. Paani direction mein ramp ki steepness batati hai ki plant har cup paani par kitna zyada lamba hota hai — maan ke ki dhoop same rehti hai — aur dhoop direction ki steepness light ke liye wahi kaam karti hai. Multiple regression bas us best ramp ko dhundhne ka maths hai.

Flashcards

Multiple regression mein partial slope

β_{j}

kya measure karta hai?

y

mein change per unit change in

x_{j}

jabki baaki saare predictors fixed hon (uska unique contribution).

Least-squares estimator ko matrix form mein likho.

\hat{β} = (X^{⊤} X)^{- 1} X^{⊤} y

Normal equations batao.

X^{⊤} X \hat{β} = X^{⊤} y

X

mein ones ka column kyun add kiya jaata hai?

Intercept

β_{0}

estimate karne ke liye; iske bina hyperplane origin se force hota hai.

Multiple regression kaunsa objective minimise karta hai?

Sum of squared residuals

∥ y - X β ∥^{2}

Geometrically,

\hat{y}

kya hai?

y

ka orthogonal projection

X

ke column space par; residual uske perpendicular hota hai.

Adjusted

R^{2}

R^{2}

se zyada kyun use karte hain?

R^{2}

kabhi kam nahi hota jab predictors add kiye jaate hain; adjusted

R^{2}

degrees of freedom ke zariye extra parameters ko penalise karta hai.

Multicollinearity ke under kya problem hoti hai?

X^{⊤} X

near-singular ho jaata hai, isliye uska inverse aur coefficient variances explode ho jaate hain → unstable estimates.

R^{2}

ko sums of squares ke zariye define karo.

R^{2} = 1 - S S E / S S T = S S R / S S T

OLS mein squared (absolute nahi) residuals kyun?

Smooth/differentiable, convex, unique closed-form solution deta hai aur orthogonal-projection geometry milti hai.

Connections

Simple Linear Regression — $p = 1$ ka special case.
Ordinary Least Squares — yahan use kiya gaya estimation principle.
Orthogonal Projection — fitted values ka geometric meaning.
Multicollinearity & VIF — jab $X^{⊤} X$ behave nahi karta.
Gauss–Markov Theorem — kyun OLS assumptions ke under BLUE hai.
Coefficient of Determination — $R^{2}$ aur adjusted $R^{2}$ .
Matrix Inverse / Positive Semidefinite Matrices — normal equations ki solvability.