Exercises — Assumptions of linear regression

WHAT we look for: a residual plot is our x-ray. A pattern means a broken assumption; no pattern means healthy. Read each against the dashed zero line.

• Plot A → none violated. A flat, even band around zero is exactly the picture of good residuals: mean zero, constant spread.
• Plot B → Linearity violated. The U-shape means the straight line is systematically too high in the middle and too low at the ends — the true relationship is curved. WHY: if $E[Y\mid X]$ were truly linear, residuals would carry no leftover shape.
• Plot C → Homoscedasticity violated (heteroscedasticity). The funnel means the spread of errors grows with $\hat{y}$, so $\text{Var}({ϵ}_i)$ is not the same constant ${\sigma }^2$ everywhere.

• $\text{Cov}({ϵ}_i,{ϵ}_j)=0$: this is uncorrelatedness — no linear relationship between the two errors. It is the weaker cousin of the Independence assumption. Strictly, zero covariance alone does not guarantee full independence; two variables can be uncorrelated yet still dependent in a nonlinear way. The two notions coincide only when the errors are jointly normal, which is exactly the setting we usually work in — under normality, $\text{Cov}=0\Rightarrow$ independence. Plain reading: one observation's error carries no linear information about another's.
• ${ϵ}_i\sim N(0,{\sigma }^2)$: Normality — each error is drawn from a bell curve centred at 0 with the same spread ${\sigma }^2$. Needed for exact $t$- and $F$-tests, not for unbiased prediction.

The tool and WHY: Multicollinearity means a predictor is predictable from the others. We measure that predictability with $R_j^2$ — the fraction of $X_j$'s variance the other predictors explain. High $R_j^2$ → $X_j$ is redundant.

${\text{VIF}}_j=\frac{1}{1-R_j^2}$ ${\text{VIF}}_3=\frac{1}{1-0.90}=\frac{1}{0.10}=10$ Decision: $\text{VIF}=10$ sits right at the classic danger threshold. Where does 10 come from? ${\text{VIF}}_j$ is the factor by which the variance of ${\hat{\beta }}_j$ is multiplied compared to an uncorrelated predictor, so the standard error is inflated by $\sqrt{{\text{VIF}}_j}$. At $\text{VIF}=10$ that is $\sqrt{10}\approx 3.16\times$ wider intervals — and $\text{VIF}=10$ corresponds to $R_j^2=0.9$, i.e. 90% of the predictor is already explained by the others. The "10" rule is an empirical convention (Kutner et al.), not a theorem: it flags the point where inflation becomes large enough to routinely destabilise signs and interpretation. Red flag — investigate/drop/combine or use ridge.

Linear model: $\hat{y}=50+0.2(500)=50+100=150$ (i.e. $150k).

Log model (natural log, base $e$): $\ln \hat{y}=4.0+0.001(500)=4.0+0.5=4.5$. To undo a natural log we apply its inverse, the exponential $e^{(\cdot )}$: $\hat{y}=e^{4.5}\approx 90.02$ WHY exponentiate: we modelled $\ln Y$, so predictions live in log-space; $e^{(\cdot )}$ is the one operation that returns us to dollar-space. The two numbers differ because the log model is a different curve fit to stabilise variance, not the same line.

The picture of $\Sigma$: healthy errors → constant variance on the diagonal (${\sigma }^2$ repeated) and zeros off-diagonal (no correlation). That is precisely $\Sigma ={\sigma }^{2}I$, the identity scaled by ${\sigma }^2$.

Substitute $\Sigma ={\sigma }^{2}I$: $(X^{T}X{)}^{-1}{X}^T({\sigma }^{2}I)X(X^{T}X{)}^{-1}.$ Pull the scalar ${\sigma }^2$ out and drop $I$: $={\sigma }^2(X^{T}X{)}^{-1}(X^{T}X)(X^{T}X{)}^{-1}={\sigma }^2(X^{T}X{)}^{-1}.$ The middle $(X^{T}X{)}^{-1}(X^{T}X)$ cancels to the identity, leaving the clean ${\sigma }^2(X^{T}X{)}^{-1}$.

What breaks otherwise: if $\Sigma$ has different ${\sigma }_i^2$ on its diagonal (heteroscedasticity) or nonzero off-diagonals (correlated errors), no cancellation happens; the sandwich stays messy. OLS still weights every point equally, so it overweights noisy points relative to the optimal weight $1/{\sigma }_i^2$ used by WLS. Result: estimates stay unbiased but become inefficient, and the reported standard errors are simply wrong.

The tool and WHY: the Durbin–Watson statistic $d$ measures whether consecutive residuals move together. $d\approx 2$ means no autocorrelation; $d\to 0$ means strong positive autocorrelation (a high residual tends to be followed by another high one); $d\to 4$ means strong negative autocorrelation (residuals flip sign each step).

The inconclusive region (the standard edge case): the test does not use a single cutoff. It compares $d$ against two tabulated critical values, a lower $d_L$ and an upper $d_U$ (both depend on $n$ and the number of predictors). The verdict is:

• $d<d_L$ → positive autocorrelation (reject independence),
• $d>d_U$ (and $d<2$) → no evidence of positive autocorrelation,
• $d_L\le d\le d_U$ → inconclusive — the test cannot decide; gather more data or use a complementary test (e.g. Breusch–Godfrey). The mirror band $4-d_U,4-d_L$ handles the negative side symmetrically.

Verdicts:

• $d=2.02$: essentially 2 → independence looks fine. Good.
• $d=0.6$: well below any typical $d_L$ → positive autocorrelation — independence violated. Standard errors too small, $p$-values too optimistic.
• $d=3.4$: above 3, i.e. beyond the mirror upper band → negative autocorrelation — independence violated in the opposite direction.

Read the Q-Q shape: a Q-Q plot puts sorted residual quantiles (y-axis) against theoretical normal quantiles (x-axis). Perfect normality → all points on the 45° line. In the figure the right tail rises above the line (residuals bigger than a normal would allow) and the left tail dips below → heavy tails on both ends, more extreme errors than a bell curve predicts.

Can you trust the $t$-test? With only $n=15$, no — not safely. The $t$-test relies on normal errors when $n$ is small. The Central Limit Theorem rescue needs a reasonably large $n$ (typically $>30$–$50$); $15$ is too small for CLT to smooth heavy tails. Fix: use bootstrap $p$-values or robust regression instead of the raw $t$-test.

Two distinct diseases:

1. Funnel → Homoscedasticity violated. Variance of income grows with education. Fix: $\ln (\text{income})$ transform or WLS with weights $1/{\sigma }_i^2$.
2. Correlation 0.95 → Multicollinearity. Schooling years and degree level carry nearly the same information ($\text{VIF}=1/(1-0.95^2)=1/(1-0.9025)\approx 10.26$). Fix: drop one, combine them, or use ridge regression.

Order matters: fix multicollinearity/structure of $X$ first (it changes which coefficients exist), then re-examine the residual spread and apply the variance fix — because transforming $Y$ or reweighting only makes sense once the predictor set is settled.

WHY weight by $1/{\sigma }_i^2$: a point with huge variance is unreliable — it should whisper, not shout, in the fit. WLS listens to each point in inverse proportion to its noise. $w_1=\frac{1}{{\sigma }_1^2}=\frac{1}{1}=1,w_2=\frac{1}{{\sigma }_2^2}=\frac{1}{100}=\mathrm{0.01.}$ Weight ratio: $\frac{w_1}{w_2}=\frac{1}{0.01}=100$. The clean point counts 100× more than the noisy one.

OLS by contrast sets all weights equal ($w_1=w_2=1$), so it treats the wildly noisy point as equally trustworthy. That equal treatment is exactly why OLS is inefficient under heteroscedasticity: it wastes information by over-listening to noise. (Estimates remain unbiased — see the L3 trap.)

The mechanism: adding $\lambda I$ shifts every eigenvalue of $X^{T}X$ up by $\lambda$. So the smallest eigenvalue moves from ${\lambda }_{\min }=0.01⟶{\lambda }_{\min }+\lambda =0.01+1=\mathrm{1.01.}$ Why this rescues us: the inverse's blow-up is governed by $1/{\lambda }_{\min }$. Before ridge that factor is $1/0.01=100$ (wild variance in $\hat{\beta }$). After ridge it is $1/1.01\approx 0.990$ — a 100× reduction in the amplification.

Bias–variance link: ridge shrinks coefficients toward zero. That introduces a small bias (we no longer target the exact OLS solution) but slashes variance (coefficients stop swinging). This is the Bias-Variance Tradeoff in action — we trade a pinch of bias for a large drop in variance, and the overall prediction error falls.