Assumptions of linear regression — Hinglish notes

2.2.7 · AI-ML › Linear & Logistic Regression

The Core Assumptions (LINE + No Multicollinearity + Normality)

1. Linearity: True Relationship Linear Hai

Yeh kyun matter karta hai: Agar true relationship $Y = X^{3} + ϵ$ hai aur hum $Y = β_{0} + β_{1} X$ fit karein, toh hum ek cubic curve pe straight line force kar rahe hain—residuals mein patterns aayenge, predictions biased ho jaayengi.

Kaise check karein: Residuals vs. fitted values plot karo. Zero ke around horizontal band = achha. Systematic curve ya U-shape = linearity violated (note: funnel shape heteroscedasticity signal karta hai, nonlinearity nahi—neeche dekho).

Fix: Predictors ko transform karo ( $lo g X$ , $X^{2}$ ), polynomial regression use karo, ya non-linear models pe switch karo (trees, neural nets).

2. Independence: Observations Independent Hain

Yeh kyun matter karta hai: Agar errors correlated hain (jaise time series mein: aaj ka temperature kal ka predict karta hai), toh standard errors artificially shrink ho jaate hain, jisse $p$ -values bahut chhote lag te hain. Tumhe lagta hai significance hai, lekin woh ek illusion hai.

Violation ka example: Stock prices, longitudinal data (ek hi insaan ka time ke saath measure karna), spatial data (kareeb ke ghar).

Kaise check karein: Durbin-Watson test (time series), residual ACF plots. Agar $d \approx 2$ hai, toh independence hold hoti hai; $d < 1$ ya $d > 3$ = dikkat.

Fix: Generalized least squares (GLS), time-series models (ARIMA), ya mixed-effects models use karo.

3. Homoscedasticity: Errors Ka Constant Variance

Derivation ki zaroorat kyun hai: OLS $\hat{β}$ ko $\sum (y_{i} - \overset{y}{^}_{i})^{2}$ minimize karke estimate karta hai. $\hat{β}$ ka variance yeh hai: $Var (\hat{β}) = (X^{T} X)^{- 1} X^{T} Σ X (X^{T} X)^{- 1}$ Agar $Σ = σ^{2} I$ (homoscedastic) hai, toh yeh $σ^{2} (X^{T} X)^{- 1}$ mein simplify ho jaata hai—clean aur computable. Agar nahi, toh $Σ$ varying $σ_{i}^{2}$ ke saath diagonal hai. Kyunki OLS har observation ko equal weight deta hai, yeh effectively noisy high-variance points ko overweight karta hai (optimal GLS weight $1/ σ_{i}^{2}$ ke relative, jo unhe down-weight karta)। Isse OLS inefficient ho jaata hai aur uske standard errors galat ho jaate hain.

Yeh kyun matter karta hai: Heteroscedasticity (non-constant variance) confidence intervals aur $p$ -values ko untrustworthy bana deti hai. Estimates unbiased rehte hain lekin inefficient ho jaate hain.

Example: Education se income predict karna. Low education = tight income range; high education = huge variance (kuch log millions kamaate hain, kuch nahi).

Kaise check karein: Residuals vs. fitted plot (dekhो funnel shape ke liye—heteroscedasticity ka tell-tale sign), Breusch-Pagan test, White test.

Fix: $Y$ ko transform karo (log, square-root), weighted least squares (WLS) use karo jahan $w_{i} = 1/ σ_{i}^{2}$ , ya robust standard errors.

4. Errors ki Normality (Inference ke liye)

Zaroori distinction:

Prediction ke liye: normality required nahi hai. OLS phir bhi BLUE (Best Linear Unbiased Estimator) hai Gauss-Markov ke according.
Hypothesis tests aur confidence intervals ke liye: hume $t$ -distributions aur $F$ -tests use karne ke liye normality chahiye. Iske bina, $p$ -values approximate hote hain.

Yeh kyun matter karta hai: Agar errors heavy-tailed hain (Cauchy, $t$ -distribution with df=2), toh kuch outliers $\hat{β}$ par dominate kar lete hain, aur intervals bahut narrow ho jaate hain.

Kaise check karein: Q-Q plot (quantile-quantile), Shapiro-Wilk test, residuals ka histogram.

Fix: Bade $n$ (>30–50) ke saath, Central Limit Theorem tumhe bachata hai— $\hat{β}$ ki sampling distribution normal approach karti hai chahe $ϵ_{i}$ normal na ho. Chhote $n$ ke liye: $Y$ transform karo, bootstrap inference use karo, ya robust regression.

5. No Multicollinearity: Predictors Perfectly Correlated Nahi Hain

Problem ki derivation: OLS $(X^{T} X) \hat{β} = X^{T} y$ solve karta hai. Agar $X$ ke columns nearly colinear hain, toh $X^{T} X$ ka determinant bahut chhota hota hai—nearly singular. Data mein chhoti si changes $\hat{β}$ mein bade swings cause karti hain.

Matrix inverse errors amplify karta hai: $\hat{β} = (X^{T} X)^{- 1} X^{T} y$ Agar $λ_{m i n} (X^{T} X) \approx 0$ hai, toh $(X^{T} X)^{- 1}$ mein entries $\propto 1/ λ_{m i n} \to \infty$ hoti hain.

Yeh kyun matter karta hai:

Coefficients ke bahut bade standard errors hote hain (wide confidence intervals)
Minor data changes ke saath signs flip ho sakte hain
Model uninterpretable ho jaata hai: " $X_{1}$ important hai ya $X_{2}$ ?" Dono same info carry karte hain, toh batana mushkil hai.

Example: House price predict karna "area in sq ft" aur "area in sq meters" dono ke saath. Perfect colinearity: $X_{2} = 0.0929 X_{1}$ .

Kaise check karein:

Variance Inflation Factor (VIF): $V I F_{j} = \frac{1}{1 - R _{j}^{2}}$ , jahan $R_{j}^{2}$ $X_{j}$ ko baaki sab predictors par regress karne se aata hai.
- VIF < 5: theek hai
- VIF > 10: serious multicollinearity
Correlation matrix heatmap, $X^{T} X$ ka condition number.

Fix: Correlated predictors mein se ek drop karo, unhe combine karo (PCA), ya ridge regression ( $L_{2}$ penalty coefficients shrink karta hai) use karo.

Summary Table

Assumption	Math	Check	Fix
Linearity	$E [Y ∣ X] = X β$	Residual plot (curve/U-shape)	$X$ ya $Y$ transform karo
Independence	$Cov (ϵ_{i}, ϵ_{j}) = 0$	Durbin-Watson, ACF	GLS, time-series model
Homoscedasticity	$Var (ϵ_{i}) = σ^{2}$	Residual plot (funnel), Breusch-Pagan	WLS, log transform
Normality	$ϵ \sim N (0, σ^{2})$	Q-Q plot	CLT (large $n$ ), bootstrap
No Multicolinearity	$det (X^{T} X) ≫ 0$	VIF	Predictors drop karo, ridge

Recall Feynman Explanation: Aise Batao Jaise Main 12 Saal Ka Hoon

Socho tum ek ruler (regression line) use karke bachon ki age se unki height guess karne ki koshish kar rahe ho. Lekin yeh tabhi kaam karta hai jab kuch rules sach hon:

Linearity: Height sach mein age ke saath seedhi-si badhti hai. Agar 10 saal mein bacche achanak chhote ho jaayein, toh tumhara ruler bekar hai.
Independence: Ek bacche ki height doosre ko affect nahi karti. Agar twins hamesha exactly match karein, toh tum double-counting kar rahe ho.
Homoscedasticity: Tumhari guess mein "error" 5-saal-walon aur 15-saal-walon ke liye same hota hai. Agar teenagers bahut vary karein (kuch tall, kuch short) lekin toddlers sab similar hon, toh tumhare ruler ki confidence toot jaati hai.
Normality: Tumhari galtiyan random bell-curve errors hain, na ki ek taraf ki (hamesha bahut chhota guess karna).
No multicollinearity: Tum "age in years" aur "age in months" saath mein use nahi kar rahe—woh toh same information hai do baar, ruler ko confuse karti hai.

Yeh rules todo, aur tumhare guesses shayad average pe close hon, lekin tumhara confidence ("Mujhe 95% yakeen hai!") ek jhooth hai.

Connections

Ordinary Least Squares (OLS): OLS tabhi BLUE hai jab Gauss-Markov assumptions hold hon (linearity, independence, homoscedasticity). Normality inference ke liye extra hai.
Residual Analysis: Diagnostic plots (residual vs. fitted, Q-Q) directly inhi assumptions ko test karte hain.
Weighted Least Squares: Heteroscedasticity fix karta hai observations ko unke variance ke inverse se weight dekar.
Ridge and Lasso Regression: Ridge multicollinearity handle karta hai correlated coefficients ko shrink karke.
Generalized Linear Models (GLMs): Linearity aur normality relax karte hain—non-normal errors allow karte hain (Poisson, binomial) aur link functions.
Hypothesis Testing in Regression: $t$ -tests aur $F$ -tests errors ki normality assume karte hain. Violated? Permutation tests ya bootstrap use karo.
Bias-Variance Tradeoff: Violations $\hat{β}$ ka variance badhate hain (wider uncertainty) lekin estimates bias nahi karte (OLS Gauss-Markov ke under abhi bhi unbiased hai).

#flashcards/ai-ml

Linear regression ki paanch core assumptions kya hain?

Linearity, Independence, Normality (of errors), Equal variance (homoscedasticity), No multicollinearity (LINE + Multi + Normal).

OLS ke liye homoscedasticity kyun matter karta hai?

Non-constant variance standard errors aur confidence intervals ko galat bana deta hai, chahe coefficient estimates unbiased rahen. OLS (equal weights) high-variance points ko optimal GLS weights ke relative overweight karta hai, jisse woh inefficient ho jaata hai.

Errors ki independence ka mathematical condition kya hai?

Cov (ϵ_{i}, ϵ_{j}) = 0

sabhi

i \neq = j

ke liye. Alag-alag observations ke errors correlate nahi karte.

Multicollinearity kaise check karte hain?

Har predictor ka Variance Inflation Factor (VIF) compute karo.

V I F_{j} = \frac{1}{1 - R _{j}^{2}}

jahan

R_{j}^{2}

X_{j}

ko doosre predictors par regress karne se aata hai. VIF > 10 serious multicollinearity indicate karta hai.

Kya OLS estimates unbiased hone ke liye errors ki normality zaruri hai?

Nahi. Gauss-Markov theorem ke according, OLS linearity, independence, aur homoscedasticity ke under unbiased hai. Normality sirf hypothesis tests aur confidence intervals ke liye chahiye. Large samples mein, CLT normality assumption ko kam critical bana deta hai.

Residual plot mein funnel shape kya indicate karta hai?

Heteroscedasticity—errors ka variance fitted values ke saath change hota hai (nonlinearity NAHI).

Y

ka log transformation ya weighted least squares se fix karo.

Heteroscedasticity ke opposite, nonlinearity kaunsa residual-plot pattern indicate karta hai?

Ek systematic curve ya U-shape (residuals ka mean zero se drift karta hai), funnel nahi. Funnel heteroscedasticity ki pehchaan hai.

Ek regression mein do predictors

X_{1}

aur

X_{2} = 2 X_{1}

kyun nahi rakh sakte?

Perfect multicollinearity

X^{T} X

ko singular bana deti hai (determinant = 0), toh

(X^{T} X)^{- 1}

exist nahi karta aur OLS unique coefficients compute nahi kar sakta.

Q-Q plot kya diagnose karta hai?

Residuals ki normality. Points ek 45-degree line follow karne chahiye. Tails mein deviations skewness ya heavy tails (non-normal errors) indicate karti hain.

Agar errors independent hain lekin normal nahi, toh inference kab valid hai?

Bade sample size (

n > 30

–

50

) ke saath, Central Limit Theorem ensure karta hai ki

\hat{β}

ki sampling distribution approximately normal ho, toh

t

-tests aur confidence intervals valid rehte hain.

Independence assumption violate hone ka kya consequence hai?

Standard errors underestimated ho jaate hain (bahut chhote), jisse inflated

t

-statistics aur hypothesis tests mein false positives aate hain. Time series aur spatial data mein common hai.