Assumptions of linear regression

The Core Assumptions (LINE + No Multicollinearity + Normality)

1. Linearity: True Relationship is Linear

Why this matters: If the true relationship is $Y = X^3 + \epsilon$ and we fit $Y = \beta_0 + \beta_1 X$ , we're forcing a straight line through a cubic—residuals will show patterns, predictions will be biased.

How to check: Plot residuals vs. fitted values. A horizontal band around zero = good. A systematic curve or U-shape = linearity violated (note: a funnel shape signals heteroscedasticity, not nonlinearity—see below).

Fix: Transform predictors ( $\log X$ , $X^2$ ), use polynomial regression, or switch to non-linear models (trees, neural nets).

2. Independence: Observations Are Independent

Why this matters: If errors are correlated (e.g., time series: today's temperature predicts tomorrow's), standard errors shrink artificially, making $p$ -values too small. You think you have significance, but it's an illusion.

Example of violation: Stock prices, longitudinal data (same person measured over time), spatial data (nearby houses).

How to check: Durbin-Watson test (time series), residual ACF plots. If $d \approx 2$ , independence holds; $d < 1$ or $d > 3$ = trouble.

Fix: Use generalized least squares (GLS), time-series models (ARIMA), or mixed-effects models.

3. Homoscedasticity: Constant Variance of Errors

Derivation of why it's needed: OLS estimates $\hat{\beta}$ by minimizing $\sum (y_i - \hat{y}_i)^2$ . The variance of $\hat{\beta}$ is: $\text{Var}(\hat{\beta}) = (X^T X)^{-1} X^T \Sigma X (X^T X)^{-1}$ If $\Sigma = \sigma^2 I$ (homoscedastic), this simplifies to $\sigma^2 (X^T X)^{-1}$ —clean and computable. If not, $\Sigma$ is diagonal with varying $\sigma_i^2$ . Because OLS gives every observation equal weight, it effectively overweights the noisy high-variance points (relative to the optimal GLS weight $1/\sigma_i^2$ , which would down-weight them). This makes OLS inefficient and its standard errors wrong.

Why this matters: Heteroscedasticity (non-constant variance) makes confidence intervals and $p$ -values untrustworthy. Estimates stay unbiased but inefficient.

Example: Predicting income from education. Low education = tight income range; high education = huge variance (some earn millions, some don't).

How to check: Residuals vs. fitted plot (look for a funnel shape—the tell-tale sign of heteroscedasticity), Breusch-Pagan test, White test.

Fix: Transform $Y$ (log, square-root), use weighted least squares (WLS) where $w_i = 1/\sigma_i^2$ , or robust standard errors.

Worked example Worked Example: Detecting Heteroscedasticity

Data: Predicting house price ( $Y$ , in $1000s) from square footage ( $X$ ). Step 1: Fit OLS: $\hat{y} = 50 + 0.2x$ .

Step 2: Compute residuals $e_i = y_i - \hat{y}_i$ . Step 3: Plot $e_i$ vs. $\hat{y}_i$ :

Small houses ($100k): residuals $\in [-5, 5]$
Large houses ($500k): residuals $\in [-50, 50]$ Why this step? The spread grows with $\hat{y}$ —variance increases with price (a funnel).

Step 4: Apply $\log(Y)$ transformation: $\log(\hat{y}) = 4.0 + 0.001x$ Now residual plot shows constant spread.

Why the fix works: Multiplication becomes addition in log-space: if $Y = \beta X \cdot \epsilon$ , then $\log Y = \log \beta + \log X + \log \epsilon$ , stabilizing variance.

4. Normality of Errors (for Inference)

Important distinction:

For prediction: normality is not required. OLS is still BLUE (Best Linear Unbiased Estimator) by Gauss-Markov.
For hypothesis tests and confidence intervals: we need normality to use $t$ -distributions and $F$ -tests. Without it, $p$ -values are approximate.

Why this matters: If errors are heavy-tailed (Cauchy, $t$ -distribution with df=2), a few outliers dominate $\hat{\beta}$ , and intervals are too narrow.

How to check: Q-Q plot (quantile-quantile), Shapiro-Wilk test, histogram of residuals.

Fix: With large $n$ (>30–50), Central Limit Theorem rescues you—sampling distribution of $\hat{\beta}$ approaches normal even if $\epsilon_i$ is not. For small $n$ : transform $Y$ , use bootstrap inference, or robust regression.

5. No Multicollinearity: Predictors Are Not Perfectly Correlated

Derivation of the problem: OLS solves $(X^T X) \hat{\beta} = X^T y$ . If columns of $X$ are nearly colinear, $X^T X$ has a tiny determinant—nearly singular. Small changes in data cause huge swings in $\hat{\beta}$ .

Matrix inverse amplifies errors: $\hat{\beta} = (X^T X)^{-1} X^T y$ If $\lambda_{\min}(X^T X) \approx 0$ , $(X^T X)^{-1}$ has entries $\propto 1/\lambda_{\min} \to \infty$ .

Why this matters:

Coefficients have huge standard errors (wide confidence intervals)
Signs can flip with minor data changes
Model becomes uninterpretable: "Is $X_1$ or $X_2$ important?" Both carry the same info, so you can't tell.

Example: Predicting house price with both "area in sq ft" and "area in sq meters". Perfect colinearity: $X_2 = 0.0929 X_1$ .

How to check:

Variance Inflation Factor (VIF): $VIF_j = \frac{1}{1- R_j^2}$ $V I F_{j} = \frac{1}{1 - R _{j}^{2}}$ , where $R_j^2$ $R_{j}^{2}$ is from regressing $X_j$ $X_{j}$ on all other predictors.
- VIF < 5: OK
- VIF > 10: serious multicollinearity
Correlation matrix heatmap, condition number of $X^T X$ .

Fix: Drop one of the corelated predictors, combine them (PCA), or use ridge regression ( $L_2$ penalty shrinks coefficients).

Summary Table

Assumption	Math	Check	Fix
Linearity	$E[Y \mid X] = X\beta$	Residual plot (curve/U-shape)	Transform $X$ or $Y$
Independence	$\text{Cov}(\epsilon_i, \epsilon_j)=0$	Durbin-Watson, ACF	GLS, time-series model
Homoscedasticity	$\text{Var}(\epsilon_i)=\sigma^2$	Residual plot (funnel), Breusch-Pagan	WLS, log transform
Normality	$\epsilon \sim N(0,\sigma^2)$	Q-Q plot	CLT (large $n$ ), bootstrap
No Multicolinearity	$\det(X^T X) \gg 0$	VIF	Drop predictors, ridge

Recall Feynman Explanation: Explain Like I'm 12

Imagine you're trying to guess how tall kids are based on their age using a ruler (regression line). But this only works if some rules are true:

Linearity: Height really does grow in a straight-ish pattern with age. If kids suddenly shrink at age 10, your ruler is useless.
Independence: Each kid's height doesn't affect another's. If twins always match exactly, you're double-counting.
Homoscedasticity: The "error" in your guess is the same for 5-year-olds and 15-year-olds. If teenagers vary wildly (some tall, some short) but toddlers are all similar, your ruler's confidence is broken.
Normality: The mistakes you make are random bell-curve errors, not one-sided (always guessing too short).
No multicollinearity: You're not using "age in years" and "age in months" together—that's the same info twice, confusing the ruler.

Break these rules, and your guesses might be close on average, but your confidence ("I'm 95% sure!") is a lie.

Connections

Ordinary Least Squares (OLS): OLS is BLUE only if Gauss-Markov assumptions hold (linearity, independence, homoscedasticity). Normality is extra for inference.
Residual Analysis: Diagnostic plots (residual vs. fitted, Q-Q) directly test these assumptions.
Weighted Least Squares: Fixes heteroscedasticity by weighting observations inversely to their variance.
Ridge and Lasso Regression: Ridge handles multicollinearity by shrinking corelated coefficients.
Generalized Linear Models (GLMs): Relax linearity and normality—allow non-normal errors (Poisson, binomial) and link functions.
Hypothesis Testing in Regression: $t$ -tests and $F$ -tests assume normality of errors. Violated? Use permutation tests or bootstrap.
Bias-Variance Tradeoff: Violations increase variance of $\hat{\beta}$ (wider uncertainty) but may not bias estimates (OLS is still unbiased under Gauss-Markov).

#flashcards/ai-ml

What are the five core assumptions of linear regression?

Linearity, Independence, Normality (of errors), Equal variance (homoscedasticity), No multicollinearity (LINE + Multi + Normal).

Why does homoscedasticity matter for OLS?

Non-constant variance makes standard errors and confidence intervals incorrect, even though coefficient estimates remain unbiased. OLS (equal weights) overweights high-variance points relative to optimal GLS weights, making it inefficient.

What is the mathematical condition for independence of errors?

\text{Cov}(\epsilon_i, \epsilon_j) = 0

for all

i \neq j

. Errors from different observations do not correlate.

How do you check for multicollinearity?

Compute Variance Inflation Factor (VIF) for each predictor.

VIF_j = \frac{1}{1 - R_j^2}

where

R_j^2

is from regressing

X_j

on other predictors. VIF > 10 indicates serious multicollinearity.

Is normality of errors required for OLS estimates to be unbiased?

No. By Gauss-Markov theorem, OLS is unbiased under linearity, independence, and homoscedasticity. Normality is only needed for hypothesis tests and confidence intervals. For large samples, CLT makes normality assumption less critical.

What does a funnel shape in a residual plot indicate?

Heteroscedasticity—variance of errors changes with fitted values (NOT nonlinearity). Fix with log transformation of

Y

or weighted least squares.

What residual-plot pattern indicates nonlinearity (as opposed to heteroscedasticity)?

A systematic curve or U-shape (the mean of residuals drifts from zero), not a funnel. A funnel is the sign of heteroscedasticity.

Why can't you have two predictors

X_1

and

X_2 = 2X_1

in a regression?

Perfect multicollinearity makes

X^T X

singular (determinant = 0), so

(X^T X)^{-1}

doesn't exist and OLS cannot compute unique coefficients.

What does a Q-Q plot diagnose?

Normality of residuals. Points should follow a 45-degree line. Deviations in tails indicate skewness or heavy tails (non-normal errors).

If errors are independent but not normal, when is inference still valid?

With large sample size (

n > 30

–

50

), Central Limit Theorem ensures sampling distribution of

\hat{\beta}

is approximately normal, so

t

-tests and confidence intervals remain valid.

What is the consequence of violating the independence assumption?

Standard errors are underestimated (too small), leading to inflated

t

-statistics and false positives in hypothesis tests. Common in time series and spatial data.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, linear regression ek bahut powerful tool hai, lekin ye koi magic wand nahi hai—ye ek "conditional tool" hai. Iska matlab hai ki jab hum $y = \beta_0 + \beta_1 x + \epsilon$ fit karte hain, toh hum ek shart rakh rahe hote hain ki data kuch specific rules follow kare. In rules ko hum assumptions kehte hain, aur inko yaad rakhne ka easy tarika hai: LINE—Linearity, Independence, Normality, aur Equal variance (homoscedasticity). Agar ye assumptions tut jaayein, toh aapke predictions unreliable ho jaate hain, confidence intervals jhoot bolne lagte hain, aur hypothesis tests aapko galat direction mein le jaate hain. Socho jaise ek contract ki fine print—use ignore karo toh saari guarantees khatam.

Ab thoda detail mein samjho. Linearity ka matlab hai ki $Y$ ka expected value predictors ka linear function hona chahiye—agar asli relationship cubic (jaise $X^3$ ) hai aur hum straight line fit kar rahe hain, toh residuals mein pattern dikhega aur predictions biased honge. Independence ka matlab hai ki har observation ka error doosre se independent ho—time series ya same person ke repeated measurements mein ye tut jaata hai, jisse standard errors artificially chhote ho jaate hain aur aapko lagta hai significance hai jabki wo bas illusion hai. Homoscedasticity ka matlab hai errors ki variance sabhi $X$ levels par constant ho—agar variance funnel shape mein badhti-ghatti hai (jaise income prediction mein high education par variance zyada), toh OLS inefficient ho jaata hai kyunki wo noisy points ko equal weight deta hai.

Ye sab kyun important hai? Kyunki checking bahut simple hai—bas residuals vs fitted values ka plot banao. Horizontal band around zero matlab sab theek; curve ya U-shape matlab linearity ka problem; aur funnel shape matlab heteroscedasticity. Aur fixes bhi seedhe hain—transform karo ( $\log X$ , $X^2$ ), weighted least squares use karo, ya robust standard errors lagao. Bottom line: assumptions ko check karna aapke model ko trustworthy banata hai. Bina inko verify kiye number nikaalna waise hi hai jaise bina foundation ke ghar banana—dikhega sahi, par bharosa nahi kar sakte.

The Core Assumptions (LINE + No Multicollinearity + Normality)

1. Linearity: True Relationship is Linear

2. Independence: Observations Are Independent

3. Homoscedasticity: Constant Variance of Errors

4. Normality of Errors (for Inference)

5. No Multicollinearity: Predictors Are Not Perfectly Correlated

Summary Table

Connections

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Linear & Logistic Regression

Connections