Multiple linear regression — Hinglish notes

2.2.2 · AI-ML › Linear & Logistic Regression

What Is Multiple Linear Regression?

Multiple features kyun? Real-world phenomena bahut saare variables pe depend karte hain. Single-feature regression confounding factors ko ignore karta hai. Multiple regression is complexity ko capture karta hai agar relationships roughly linear hain.

Matrix Notation: Kyun Chahiye Yeh

Har prediction ke liye $\overset{y}{^} = β_{0} + β_{1} x_{1} + β_{2} x_{2} + \dots$ likhna tedious hai. $m$ training examples aur $n$ features ke saath, hamare paas $m$ equations hain. Matrix form ise elegantly compress kar deta hai.

Step 1: Data ko matrices ke roop mein represent karo

$m$ examples ke liye, har ek ke saath $n$ features: $X = 11 ⋮ 1 x_{1}^{(1)} x_{1}^{(2)} ⋮ x_{1}^{(m)} x_{2}^{(1)} x_{2}^{(2)} ⋱ x_{2}^{(m)} \dots \dots ⋮ \dots x_{n}^{(1)} x_{n}^{(2)} x_{n}^{(m)}, β = β_{0} β_{1} ⋮ β_{n}, y = y^{(1)} y^{(2)} ⋮ y^{(m)}$

Leading 1's column kyun? Yeh $β_{0}$ ka "feature" hai. Intercept ko sirf ek aur coefficient bana deta hai dot product mein.

Step 2: Predictions as matrix multiplication

$\hat{y} = X β$

Yeh single equation $m$ alag-alag prediction formulas replace kar deta hai. $X$ ki har row ko $β$ se dot karo to ek prediction milti hai.

Optimal Coefficients Derive Karna: The Normal Equation

Goal: Woh $β$ dhundho jo mean squared error (MSE) minimize kare: $J (β) = \frac{1}{m} \sum_{i = 1}^{m} (y^{(i)} - \overset{y}{^}^{(i)})^{2}$

Yeh step kyun? Hum woh jagah dhundh rahe hain jahan cost function ka slope zero ho—minimum. Doosra derivative (Hessian $X^{⊤} X$ ) positive definite hai, jo confirm karta hai ki yeh minimum hai, maximum nahi.

Normal Equation Kab Fail Karta Hai?

Worked Examples

Worked example Example 1: House Price Prediction (2 Features)

Data: 3 houses, size (sq ft) aur bedrooms ke saath

Size $(x_{1})$	Bedrooms $(x_{2})$	Price $(y)$ (k$)
1500	3	300
2000	4	400
1800	3	350

Matrices setup karo: $X = 111150020001800343, y = 300400350$

Step 1: $X^{⊤} X$ compute karo

Pehle $X^{⊤}$ sahi se likho (ek $3 \times 3$ transpose, har row mein 3 entries): $X^{⊤} = 115003120004118003$

Ab multiply karo. Entry $(2, 2)$ hai $150 0^{2} + 200 0^{2} + 180 0^{2} = 2, 250, 000 + 4, 000, 000 + 3, 240, 000 = 9, 490, 000$ : $X^{⊤} X = 3530010 5300 9, 490, 000 17900 101790034$

Yeh step kyun? Hum saare examples ke features ke beech dot products accumulate kar rahe hain. Yeh matrix encode karta hai ki features kaise correlate karte hain.

Step 2: $X^{⊤} y$ compute karo

Entry 2 hai $1500 \times 300 + 2000 \times 400 + 1800 \times 350 = 450, 000 + 800, 000 + 630, 000 = 1, 880, 000$ : $X^{⊤} y = 115003120004118003300400350 = 1050 1, 880, 000 3550$

Yeh step kyun? Yeh features aur target ke beech correlation hai. Yeh batata hai ki "har feature target ko kitna pull karta hai?"

Step 3: Solve karo $β = (X^{⊤} X)^{- 1} X^{⊤} y$

⚠️ Note: Sirf 3 examples aur 3 unknowns ( $β_{0}, β_{1}, β_{2}$ ) ke saath, yeh system exactly determined hai — plane teeno points se zero residual ke saath guzarta hai. Linear system $X^{⊤} X β = X^{⊤} y$ solve karne par (SymPy se verify kiya neeche) milta hai: $β = β_{0} β_{1} β_{2} = - 150 0.2 50$

Interpretation: Price = $- 150 k +$ 0.2/sqft × size + $50k × bedrooms

House 1 check (1500 sqft, 3 bed): $\overset{y}{^} = - 150 + 0.2 (1500) + 50 (3) = - 150 + 300 + 150 = 300 k ✓$

Naye house ke liye prediction (1600 sqft, 3 bed): $\overset{y}{^} = - 150 + 0.2 (1600) + 50 (3) = - 150 + 320 + 150 = 320 k$

(Negative intercept mathematically theek hai—yeh ek extrapolation artifact hai, koi real "zero size pe price" nahi.)

Assumptions of Multiple Linear Regression

Normal Equation meaningful results dene ke liye, hum assume karte hain:

Linearity: True relationship linear hai (ya almost linear)
Independence: Observations independent hain (koi time-series autocorrelation nahi)
Homoscedasticity: Error variance saare $x$ values mein constant hai
No perfect multicollinearity: Features ek doosre ke exact linear combos nahi hain
Normality of errors (inference ke liye): Errors $\sim N (0, σ^{2})$ hain confidence intervals ke liye

Agar violate ho toh kya hoga?

Non-linearity → poor fit, polynomial features add karo ya non-linear model use karo
Multicollinearity → unstable coefficients, high variance
Heteroscedasticity → confidence intervals galat, weighted least squares use karo

Computational Complexity

Normal Equation: Matrix inversion ke liye $O (n^{3})$ + $X^{⊤} X$ ke liye $O (m n^{2})$ . Dominant cost: inversion.

Kab use karo:

Chota $n$ (< 10,000 features): Normal Equation fast, exact, no hyperparameters
Bada $n$ : Gradient descent use karo ( $k$ iterations ke liye $O (k mn)$ , $k ≪ n$ )

Recall 12-Saal Ke Bachche Ko Samjhao

Socho tum guess kar rahe ho ki koi party mein kitni candy khayega. Tumne notice kiya ki teen cheezein matter karti hain: kitna bhookha hai, sweets kitne pasand hain, aur kitne dost paas hain.

Simple regression aisa hai jaise kehna "main sirf bhookh dekhuga aur baaki sab ignore karunga." Yeh silly hai! Multiple regression zyada smart hai: tum ek formula banate ho jo TEENON clues use karta hai, har ek ko ek "weight" deta hai based on kitna important hai.

Toh tumhara formula ho sakta hai: Candy khaya = 5 + 2×(bhookh) + 3×(sweets love) + 1×(dost paas). Computer sabse best weights dhundta hai purani parties dekh kar aur yeh figure out karke ki kaun se numbers sabse kam galat guesses karte hain. Yeh aisa hai jaise ek cheat sheet ho jo ek ki jagah multiple hints use karta hai!

Connections

Simple Linear Regression: Multiple regression $n$ features tak generalize karta hai
Gradient Descent: Alternative optimization jab Normal Equation expensive ho
Ridge Regression: Multicollinearity handle karne ke liye $L_{2}$ regularization add karta hai
Polynomial Regression: Multiple regression jisme $x^{2}, x^{3}, \dots$ extra features hain
Feature Scaling: Critical preprocessing—coefficients ko magnitude mein comparable rakhta hai
R-squared and Adjusted R-squared: Fit quality evaluate karne ke metrics
Overfitting: Zyada features → noise fit karne ka risk, regularization chahiye

#flashcards/ai-ml

Multiple linear regression predictions ka matrix form kya hai? :: $\hat{y} = X β$ , jahan $X$ design matrix hai (intercept ke liye leading 1's ke saath), $β$ coefficient vector hai.

Multiple linear regression ke liye Normal Equation kya hai?

β = (X^{⊤} X)^{- 1} X^{⊤} y

. MSE ka gradient zero set karke derive kiya jata hai.

Feature matrix

X

mein 1's ka column kyun add karte hain?

Intercept term

β_{0}

ko matrix multiplication

X β

ka part banana ke liye. 1's bias term ke liye "feature" ka kaam karte hain.

X^{⊤} X

kab non-invertible hota hai?

Jab (1)

m < n

(features se kam examples), ya (2) perfect multicollinearity ho (ek feature doosron ka exact linear combination ho).

Jab Normal Equation fail kare toh do alternatives kya hain?

(1) Gradient descent use karo (inversion ki zaroorat nahi), (2) Ridge regression jaisi regularization add karo (

X^{⊤} X + λ I

Multiple linear regression mein negative coefficient

β_{i}

ka kya matlab hai?

Feature

x_{i}

mein har 1-unit increase ke liye, predicted target

\overset{y}{^}

mein

∣ β_{i} ∣

units ki kami aati hai, doosre saare features ko constant rakhte hue.

Multicollinearity kya hai?

Jab features ek doosre se highly correlated hoon (jaise

x_{3} \approx 2 x_{1} + x_{2}

). Isse coefficient estimates unstable ho jate hain aur variance badh jata hai.

Normal Equation

O (n^{3})

complexity kyun hai?

Matrix inversion

(X^{⊤} X)^{- 1}

n \times n

matrix ke liye

O (n^{3})

hai. Yahi cost ko dominate karta hai.

Normal Equation ki jagah gradient descent kab use karna chahiye?

Jab

n

bada ho (> 10,000 features). Gradient descent

k

iterations ke liye

O (k mn)

hai,

O (n^{3})

inversion cost se bachata hai.

Coefficient interpretation mein "doosre features ko constant rakhna" ka kya matlab hai?

Coefficient

β_{i}

x_{i}

\overset{y}{^}

pe isolated effect measure karta hai, jaise baaki saare features freeze hain. Multiple regression har feature ka contribution alag karta hai.