Visual walkthrough — Assumptions of linear regression

This walkthrough targets the homoscedasticity assumption (assumption 3 of the parent). By the end you will see why $\text{Var}(\hat{\beta })={\sigma }^2(X^{T}X{)}^{-1}$ is the "clean" case and why the funnel-shaped residual plot is a warning sign.

---

Step 1 — What a fitted line even is

WHAT. We have dots on a plane. The horizontal position of a dot is its predictor $x$ (say house size), the vertical position is its response $y$ (say price). A fit is a straight line we lay through the cloud.

WHY. Before we can talk about "errors" or "variance of errors", we must first agree on what the line is and what a residual is. Nothing else makes sense until this picture exists.

PICTURE. In the figure, each grey dot is a data point $(x_i,y_i)$. The lavender line is our guess $\hat{y}={\hat{\beta }}_0+{\hat{\beta }}_{1}x$. The short coral stick from a dot down to the line is the residual $e_i$.

The residual is the coral stick's signed length. That stick is the hero of this entire page.

---

Step 2 — OLS: pick the line that makes the sticks shortest

WHAT. Ordinary Least Squares (OLS) chooses ${\hat{\beta }}_0,{\hat{\beta }}_1$ to make the sum of squared residuals as small as possible.

WHY squared, not the raw sticks? Two reasons, both visible in the picture. (1) A raw stick can be negative; adding negatives and positives lets a bad line hide (they cancel). Squaring makes every stick a positive area, so nothing cancels. (2) Squaring punishes a long stick far more than a short one, so the line is pulled hard toward far-away dots.

PICTURE. Each coral stick becomes a coral square whose side equals the stick length. OLS slides and tilts the line until the total pastel-square area is minimal.

The crucial fine print: notice every square counts the same. A stick of length 2 near the left contributes exactly as much per unit as a stick of length 2 on the right. OLS has no idea some dots are noisier than others. Remember this — it is the whole reason homoscedasticity matters.

---

Step 3 — Where randomness lives: the error term

WHAT. We now say the dots were generated by a rule plus noise: $y_i={\beta }_0+{\beta }_1{x}_i+{ϵ}_i.$

WHY. Real dots never sit exactly on any line. The wobble ${ϵ}_i$ ("epsilon", the true error) is the invisible dice-roll that knocks dot $i$ off the true line. We can never see ${ϵ}_i$ itself — only its shadow, the residual $e_i$ from Step 1. Understanding the spread of these dice-rolls is the key to trusting our conclusions.

PICTURE. At three x-positions we draw a small vertical bell curve (a probability shape) centred on the true line. The dot lands somewhere inside its bell. A wide bell means "this x is noisy"; a narrow bell means "this x is quiet".

---

Step 4 — The two worlds: equal bells vs. fanning bells

WHAT. Two scenarios split here.

• Homoscedastic: every bell has the same width, ${\sigma }_i^2={\sigma }^2$ for all $i$. One shared $\sigma$, no subscript.
• Heteroscedastic: the bells fan out, ${\sigma }_i^2$ grows (or shrinks) with $x$.

WHY it matters visually. When you drop the true line and look only at the residual sticks, equal bells give a flat band of residuals; fanning bells give a funnel — exactly the tell-tale shape the parent note warns about.

PICTURE. Left panel: equal-width bells → residuals form a horizontal band. Right panel: widening bells → residuals form a coral funnel.

This is the same funnel the parent's Breusch–Pagan / residual-plot check hunts for. See Residual Analysis for the diagnostic drill.

---

Step 5 — Packing everything into matrices

WHAT. To reach the uncertainty formula we stack all $n$ equations into one matrix line: $\mathbf{y}=X\mathbf{\beta }+\mathbf{ϵ}.$

WHY. With many dots (and later many predictors) writing $n$ separate equations is hopeless. A matrix is just a grid that holds them all at once, so one clean equation replaces the whole pile.

PICTURE. We draw the grid: $X$ is a tall table (one row per dot; first column all 1's for the intercept, second column the $x_i$'s), $\mathbf{\beta }$ is the short list of unknowns, $\mathbf{y}$ is the tall list of responses.

Now the whole variance story lives in one object, the covariance matrix $\Sigma$: $\Sigma =\text{Var}(\mathbf{ϵ})=\left[\begin{array}{ccc}{\sigma }_1^2& 0& \cdots \\ 0& {\sigma }_2^2& \cdots \\ ⋮& & \ddots \end{array}\right].$

• The diagonal entries are the bell-widths ${\sigma }_i^2$ — one per dot.
• The off-diagonal zeros say errors don't talk to each other (that is the independence assumption; here we assume it so we can isolate homoscedasticity).

---

Step 6 — The uncertainty of the fit: the sandwich

WHAT. OLS's solution is $\hat{\mathbf{\beta }}=(X^{T}X{)}^{-1}{X}^T\mathbf{y}$. Pushing the randomness of $\mathbf{ϵ}$ through this gives the variance of our estimated coefficients: $\text{Var}(\hat{\mathbf{\beta }})=(X^{T}X{)}^{-1}{X}^T\Sigma X(X^{T}X{)}^{-1}.$

WHY we need this. $\hat{\mathbf{\beta }}$ changes every time nature re-rolls the dice. This formula tells us how much the line would wobble if we re-collected data — that wobble becomes our confidence intervals and $p$-values (see Hypothesis Testing in Regression).

PICTURE. We draw the formula as a literal sandwich: two identical "bread" slices $(X^{T}X{)}^{-1}{X}^T$ and $X(X^{T}X{)}^{-1}$ with the filling $\Sigma$ in the middle. The filling is the only place the bell-widths enter.

If the filling $\Sigma$ is simple, the sandwich collapses. That is the next step.

---

Step 7 — The magic collapse (homoscedastic case)

WHAT. Assume homoscedasticity: every bell equal, so $\Sigma ={\sigma }^{2}I$ (the identity matrix $I$ scaled by the single number ${\sigma }^2$). Pull the scalar out: $\text{Var}(\hat{\mathbf{\beta }})=(X^{T}X{)}^{-1}{X}^T({\sigma }^{2}I)X(X^{T}X{)}^{-1}={\sigma }^2(X^{T}X{)}^{-1}.$

WHY it's beautiful. The filling ${\sigma }^{2}I$ commutes and cancels: $X^{T}IX=X^{T}X$, and one $(X^{T}X{)}^{-1}$ eats the middle $X^{T}X$, leaving a single $(X^{T}X{)}^{-1}$. The scary sandwich becomes one slice times one number — exactly the parent's "clean and computable" result.

PICTURE. The sandwich visually deflates: the identity filling lets the two breads merge into a single lavender slab, ${\sigma }^2(X^{T}X{)}^{-1}$.

---

Step 8 — When it fails: unequal bells, and the wasted information

WHAT. If bells fan out, $\Sigma \ne {\sigma }^{2}I$; the sandwich does not collapse. Worse, OLS's equal-weighting (Step 2) is now provably suboptimal: it treats a fat-bell (noisy) dot and a thin-bell (reliable) dot as equals.

WHY it's wrong. The best you can do is trust reliable dots more. The optimal weight is $w_i=1/{\sigma }_i^2$: down-weight the noisy fat-bell points. Weighted Least Squares does exactly this; OLS does not, so OLS is inefficient and its plain standard errors lie.

PICTURE. Two panels. Left: OLS gives every coral square the same "importance halo". Right: WLS shrinks the halo on fat-bell dots and enlarges it on thin-bell dots — the line now leans toward the trustworthy points.

The fixes (from the parent): transform $Y$ (e.g. $\log Y$) to flatten the funnel, use Weighted Least Squares, or use robust standard errors. When the response's variance-mean link is structural (counts, proportions), step up to Generalized Linear Models (GLMs).

---

The one-picture summary

This final figure walks left→right: raw dots → residual sticks → the funnel test → the two paths (equal bells collapse the sandwich; unequal bells demand weights).

Recall Feynman retelling — say it back in plain words

We drew dots and a line. The gap from each dot to the line is a residual stick. OLS shrinks the total area of those sticks, treating every stick as equally important. But nature adds each dot a random wobble drawn from a little bell curve. If all the bells are the same width (homoscedastic), the residuals form a flat band and the messy formula for how much our line wobbles collapses into one clean piece, ${\sigma }^2(X^{T}X{)}^{-1}$ — the noise part and the geometry part cleanly separate. If the bells fan out (heteroscedastic), the residuals form a funnel, the formula stays a messy sandwich, and OLS's "everyone equal" rule wrongly trusts the noisy fat-bell dots as much as the quiet reliable ones. The cure is to weight each dot by $1/{\sigma }_i^2$ — down-weight the noisy ones — which is exactly weighted least squares. And beware: if your x's barely vary, the geometry piece $(X^{T}X{)}^{-1}$ explodes on its own, which is the multicollinearity problem.

Recall Quick self-checks

What makes the OLS variance formula collapse from a sandwich to ${\sigma }^2(X^{T}X{)}^{-1}$? ::: The homoscedastic assumption $\Sigma ={\sigma }^{2}I$, which lets the scalar factor out and one $(X^{T}X{)}^{-1}$ cancel the middle $X^{T}X$. Why does OLS become inefficient under heteroscedasticity? ::: It weights every residual equally, but the optimal weight is $1/{\sigma }_i^2$, so OLS over-trusts high-variance points. What shape in the residual-vs-fitted plot signals heteroscedasticity? ::: A funnel (fanning) shape, not a flat band. Even with perfect homoscedasticity, what still blows up $\text{Var}(\hat{\beta })$? ::: Near-singular $X^{T}X$ (multicollinearity), since $(X^{T}X{)}^{-1}$ has entries $\propto 1/{\lambda }_{\min }$.

Related: Assumptions of linear regression (parent) · Bias-Variance Tradeoff · 2.2.07 Assumptions of linear regression (Hinglish)