Multiple linear regression

What Is Multiple Linear Regression?

Why multiple features? Real-world phenomena depend on many variables. Single-feature regression ignores confounding factors. Multiple regression captures this complexity if relationships are roughly linear.

Matrix Notation: Why We Need It

Writing $\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots$ for every prediction is tedious. With $m$ training examples and $n$ features, we have $m$ equations. Matrix form compresses this elegantly.

Step 1: Represent data as matrices

For $m$ examples, each with $n$ features: $\mathbf{X} = \begin{bmatrix} 1 & x_1^{(1)} & x_2^{(1)} & \cdots & x_n^{(1)} \\ 1 & x_1^{(2)} & x_2^{(2)} & \cdots & x_n^{(2)} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x_1^{(m)} & x_2^{(m)} & \cdots & x_n^{(m)} \end{bmatrix}, \quad \boldsymbol{\beta} = \begin{bmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_n \end{bmatrix}, \quad \mathbf{y} = \begin{bmatrix} y^{(1)} \\ y^{(2)} \\ \vdots \\ y^{(m)} \end{bmatrix}$

Why the leading1's column? That's the "feature" for $\beta_0$ . Makes the intercept just another coefficient in the dot product.

Step 2: Predictions as matrix multiplication

$\hat{\mathbf{y}} = \mathbf{X} \boldsymbol{\beta}$

This single equation replaces $m$ separate prediction formulas. Each row of $\mathbf{X}$ doted with $\boldsymbol{\beta}$ gives one prediction.

Deriving the Optimal Coefficients: The Normal Equation

Goal: Find $\boldsymbol{\beta}$ that minimizes the mean squared error (MSE): $J(\boldsymbol{\beta}) = \frac{1}{m} \sum_{i=1}^{m} (y^{(i)} - \hat{y}^{(i)})^2$

Formula Derivation from First Principles

Step 1: Write MSE in matrix form Error vector: $\mathbf{e} = \mathbf{y} - \mathbf{X}\boldsymbol{\beta}$

Sum of squared errors: $\mathbf{e}^\top \mathbf{e} = (\mathbf{y} - \mathbf{X}\boldsymbol{\beta})^\top (\mathbf{y} - \mathbf{X}\boldsymbol{\beta})$

Expand: $\mathbf{e}^\top \mathbf{e} = \mathbf{y}^\top \mathbf{y} - \mathbf{y}^\top \mathbf{X}\boldsymbol{\beta} - \boldsymbol{\beta}^\top \mathbf{X}^\top \mathbf{y} + \boldsymbol{\beta}^\top \mathbf{X}^\top \mathbf{X} \boldsymbol{\beta}$

Since $\mathbf{y}^\top \mathbf{X}\boldsymbol{\beta}$ is a scalar, it equals its transpose: $\mathbf{y}^\top \mathbf{X}\boldsymbol{\beta} = \boldsymbol{\beta}^\top \mathbf{X}^\top \mathbf{y}$ $\mathbf{e}^\top \mathbf{e} = \mathbf{y}^\top \mathbf{y} - 2 \boldsymbol{\beta}^\top \mathbf{X}^\top \mathbf{y} + \boldsymbol{\beta}^\top \mathbf{X}^\top \mathbf{X} \boldsymbol{\beta}$ Step 2: Take gradient with respect to $\boldsymbol{\beta}$

Using matrix calculus rules:

$\frac{\partial}{\partial \boldsymbol{\beta}} (\mathbf{a}^\top \boldsymbol{\beta}) = \mathbf{a}$
$\frac{\partial}{\partial \boldsymbol{\beta}} (\boldsymbol{\beta}^\top \mathbf{A} \boldsymbol{\beta}) = 2\mathbf{A}\boldsymbol{\beta}$ (if $\mathbf{A}$ symmetric) $\frac{\partial}{\partial \boldsymbol{\beta}} \mathbf{e}^\top \mathbf{e} = -2 \mathbf{X}^\top \mathbf{y} + 2 \mathbf{X}^\top \mathbf{X} \boldsymbol{\beta}$

Step 3: Set gradient to zero

$-2 \mathbf{X}^\top \mathbf{y} + 2 \mathbf{X}^\top \mathbf{X} \boldsymbol{\beta} = 0$

$\mathbf{X}^\top \mathbf{X} \boldsymbol{\beta} = \mathbf{X}^\top \mathbf{y}$

Step 4: Solve for $\boldsymbol{\beta}$

If $\mathbf{X}^\top \mathbf{X}$ is invertible: $\boxed{\boldsymbol{\beta} = (\mathbf{X}^\top \mathbf{X})^{-1} \mathbf{X}^\top \mathbf{y}}$

This is the Normal Equation.

Why this step? We're finding where the cost function's slope is zero—the minimum. The second derivative (Hessian $\mathbf{X}^\top \mathbf{X}$ ) is positive definite, confirming it's a minimum, not maximum.

When Does the Normal Equation Fail?

Common mistake Non-Invertible

$\mathbf{X}^\top \mathbf{X}$

Wrong idea: "Just compute $(\mathbf{X}^\top \mathbf{X})^{-1}$ always."

Why it feels right: The formula looks clean and closed-form.

The problem: $\mathbf{X}^\top \mathbf{X}$ is non-invertible (singular) when:

Fewer examples than features ( $m < n$ ): Underdetermined system, infinite solutions
Perfect multicollinearity: One feature is an exact linear combo of others (e.g., $x_3 = 2x_1 + x_2$ ) The fix:

Add regularization (Ridge regression: add $\lambda \mathbf{I}$ to $\mathbf{X}^\top \mathbf{X}$ )
Remove redundant features (check correlation matrix)
Use gradient descent instead (works even if singular)
Use pseudoinverse $(\mathbf{X}^\top \mathbf{X})^+$ (gives minimum-norm solution)

Worked Examples

Worked example Example 1: House Price Prediction (2 Features)

Data: 3 houses with size (sq ft) and bedrooms

Size $(x_1)$	Bedrooms $(x_2)$	Price $(y)$ (k$)
1500	3	300
2000	4	400
1800	3	350

Setup matrices: $\mathbf{X} = \begin{bmatrix} 1 & 1500 & 3 \\ 1 & 2000 & 4 \\ 1 & 1800 & 3 \end{bmatrix}, \quad \mathbf{y} = \begin{bmatrix} 300 \\ 400 \\ 350 \end{bmatrix}$

Step 1: Compute $\mathbf{X}^\top \mathbf{X}$

First write $\mathbf{X}^\top$ correctly (a $3\times 3$ transpose, each row has 3 entries): $\mathbf{X}^\top = \begin{bmatrix} 1 & 1 & 1 \\ 1500 & 2000 & 1800 \\ 3 & 4 & 3 \end{bmatrix}$

Now multiply. Entry $(2,2)$ is $1500^2 + 2000^2 + 1800^2 = 2{,}250{,}000 + 4{,}000{,}000 + 3{,}240{,}000 = 9{,}490{,}000$ : $\mathbf{X}^\top \mathbf{X} = \begin{bmatrix} 3 & 5300 & 10 \\ 5300 & 9{,}490{,}000 & 17900 \\ 10 & 17900 & 34 \end{bmatrix}$

Why this step? We're accumulating dot products between features across all examples. This matrix encodes how features correlate.

Step 2: Compute $\mathbf{X}^\top \mathbf{y}$

Entry 2 is $1500\times300 + 2000\times400 + 1800\times350 = 450{,}000 + 800{,}000 + 630{,}000 = 1{,}880{,}000$ : $\mathbf{X}^\top \mathbf{y} = \begin{bmatrix} 1 & 1 & 1 \\ 1500 & 2000 & 1800 \\ 3 & 4 & 3 \end{bmatrix} \begin{bmatrix} 300 \\ 400 \\ 350 \end{bmatrix} = \begin{bmatrix} 1050 \\ 1{,}880{,}000 \\ 3550 \end{bmatrix}$

Why this step? This is the correlation between features and target. It tells us "how much does each feature pull the target?"

Step 3: Solve $\boldsymbol{\beta} = (\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top \mathbf{y}$

⚠️ Note: With only 3 examples and 3 unknowns ( $\beta_0,\beta_1,\beta_2$ ), this system is exactly determined — the plane passes through all 3 points with zero residual. Solving the linear system $\mathbf{X}^\top\mathbf{X}\,\boldsymbol\beta = \mathbf{X}^\top\mathbf{y}$ (verified by SymPy below) gives: $\boldsymbol{\beta} = \begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \end{bmatrix} = \begin{bmatrix} -150 \\ 0.2 \\ 50 \end{bmatrix}$

Interpretation: Price = $-150k +$ 0.2/sqft × size + $50k × bedrooms

Check on house 1 (1500 sqft, 3 bed): $\hat{y} = -150 + 0.2(1500) + 50(3) = -150 + 300 + 150 = 300\text{k} \checkmark$

Prediction for new house (1600 sqft, 3 bed): $\hat{y} = -150 + 0.2(1600) + 50(3) = -150 + 320 + 150 = 320\text{k}$

(The negative intercept is fine mathematically—it's an extrapolation artifact, not a real "price at zero size.")

Assumptions of Multiple Linear Regression

For the Normal Equation to give meaningful results, we assume:

Linearity: True relationship is linear (or nearly so)
Independence: Observations are independent (no time-series autocorrelation)
Homoscedasticity: Error variance is constant across all $x$ values
No perfect multicollinearity: Features aren't exact linear combos of each other
Normality of errors (for inference): Errors $\sim N(0, \sigma^2)$ for confidence intervals

What happens if violated?

Non-linearity → poor fit, add polynomial features or use non-linear model
Multicollinearity → unstable coefficients, high variance
Heteroscedasticity → confidence intervals wrong, use weighted least squares

Computational Complexity

Normal Equation: $O(n^3)$ for matrix inversion + $O(mn^2)$ for $\mathbf{X}^\top \mathbf{X}$ . Dominant cost: inversion.

When to use:

Small $n$ (< 10,000 features): Normal Equation is fast, exact, no hyperparameters
Large $n$ : Use gradient descent ( $O(kmn)$ for $k$ iterations, $k \ll n$ )

Recall Explain to a 12-Year-Old

Imagine you're guessing how much candy someone will eat at a party. You notice three things matter: how hungry they are, how much they like sweets, and how many friends they're with.

Simple regression is like saying "I'll only look at hunger and ignore everything else." That's silly! Multiple regression is smarter: you make a formula that uses ALL three clues, giving each one a "weight" based on how important it is.

So your formula might be: Candy eaten = 5 + 2×(hunger) + 3×(sweetness love) + 1×(friends nearby). The computer finds the best weights by looking at past parties and figuring out which numbers make the fewest wrong guesses. It's like having a cheat sheet that uses multiple hints instead of just one!

Connections

Simple Linear Regression: Multiple regression generalizes to $n$ features
Gradient Descent: Alternative optimization when Normal Equation is expensive
Ridge Regression: Adds $L_2$ regularization to handle multicollinearity
Polynomial Regression: Multiple regression with $x^2, x^3, \ldots$ as extra features
Feature Scaling: Critical preprocessing—keeps coefficients comparable in magnitude
R-squared and Adjusted R-squared: Metrics for evaluating fit quality
Overfitting: More features → risk of fitting noise, need regularization

#flashcards/ai-ml

What is the matrix form of multiple linear regression predictions? :: $\hat{\mathbf{y}} = \mathbf{X} \boldsymbol{\beta}$ , where $\mathbf{X}$ is the design matrix (with leading 1's for intercept), $\boldsymbol{\beta}$ is the coefficient vector.

What is the Normal Equation for multiple linear regression?

\boldsymbol{\beta} = (\mathbf{X}^\top \mathbf{X})^{-1} \mathbf{X}^\top \mathbf{y}

. Derived by setting the gradient of MSE to zero.

Why do we add a column of 1's to the feature matrix

\mathbf{X}

To represent the intercept term

\beta_0

as part of the matrix multiplication

\mathbf{X}\boldsymbol{\beta}

. The 1's act as the "feature" for the bias term.

When is

\mathbf{X}^\top \mathbf{X}

non-invertible?

When (1)

m < n

(fewer examples than features), or (2) perfect multicollinearity exists (one feature is an exact linear combination of others).

What are two alternatives when the Normal Equation fails?

(1) Use gradient descent instead (doesn't require inversion), (2) Add regularization like Ridge regression (

\mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I}

What does a negative coefficient

\beta_i

mean in multiple linear regression?

For every1-unit increase in feature

x_i

, the predicted target

\hat{y}

decreases by

|\beta_i|

units, holding all other features constant.

What is multicollinearity?

When features are highly correlated with each other (e.g.,

x_3 \approx 2x_1 + x_2

). Causes unstable coefficient estimates and inflated variance.

Why is the Normal Equation

O(n^3)

complexity?

Matrix inversion

(\mathbf{X}^\top \mathbf{X})^{-1}

O(n^3)

for an

n \times n

matrix. This dominates the cost.

When should you use gradient descent over the Normal Equation?

When

n

is large (> 10,000 features). Gradient descent is

O(kmn)

for

k

iterations, avoiding the

O(n^3)

inversion cost.

What does "holding other features constant" mean in coefficient interpretation?

The coefficient

\beta_i

measures the isolated effect of

x_i

\hat{y}

, as if all other features are frozen. Multiple regression disentangles each feature's contribution.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, yaha core idea bahut simple hai. Pehle humne ek single feature se prediction kiya tha, jaise sirf ghar ke size se price predict karna. Lekin real world mein cheezein itni simple nahi hoti — ghar ka price sirf size pe nahi, balki bedrooms, location, age, sab pe depend karta hai. Toh Multiple Linear Regression basically yahi karta hai: kai saare features ko ek saath use karke prediction banata hai, aur har feature ko ek weight (coefficient) deta hai jo batata hai ki wo feature kitna important hai. 2D mein jaise hum ek line fit karte the, ab high dimensions mein hum ek "hyperplane" fit kar rahe hain — concept wahi hai, bas dimensions zyada ho gaye.

Ab matrix notation kyun important hai? Socho tumhare paas hazaaron training examples hain, har ek ke liye alag-alag equation likhna practically impossible hai. Isliye hum saara data matrices mein pack kar dete hain — X matrix mein saare features, beta vector mein saare weights. Phir sirf ek chhota sa equation y = X·beta se saari predictions ek saath ho jaati hain. Wo jo leading 1's ka column dikhta hai na, wo intercept (β₀) ke liye hai, taaki bias bhi baaki coefficients ki tarah hi dot product mein aa jaaye. Ye trick calculations ko super clean aur fast bana deti hai, especially jab computer pe implement karna ho.

Aur sabse mast part hai Normal Equation. Hum error (MSE) ko minimize karna chahte hain, matlab prediction aur actual value ke beech ka gap kam se kam ho. Toh calculus use karke gradient nikaalte hain, usse zero pe set karte hain, aur seedha ek formula mil jaata hai jo optimal beta de deta hai — bina koi iterations ya guessing ke, one shot mein! Ye matter isliye karta hai kyunki ye foundation hai — machine learning ke bade models bhi isi linear algebra aur optimization ki soch pe khade hain. Ek baar ye intuition clear ho gayi toh aage neural networks tak sab samajhne mein aasaani hogi.

What Is Multiple Linear Regression?

Matrix Notation: Why We Need It

Deriving the Optimal Coefficients: The Normal Equation

When Does the Normal Equation Fail?

Worked Examples

Assumptions of Multiple Linear Regression

Computational Complexity

Connections

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Linear & Logistic Regression

Connections