Simple linear regression model: Question bank

This is a rapid-fire self-test for the parent topic. Each line is a question ::: answer reveal. Cover the right side, say your reasoning out loud, then check. The answers are never a bare "yes/no" — they carry the why, because that is the part exams and interviews actually probe.

True or false — justify

True or false: OLS minimizes the sum of vertical distances from points to the line.

False. It minimizes the sum of squared vertical distances (RSS). Plain distances would let a

+ 5

and a

- 5

error cancel to zero, hiding a bad fit; squaring keeps every error positive and punishes big misses harder.

True or false: The OLS best-fit line always passes through the point

(\overset{x}{ˉ}, \overset{y}{ˉ})

True. The intercept formula

β_{0} = \overset{y}{ˉ} - β_{1} \overset{x}{ˉ}

rearranges to

\overset{y}{ˉ} = β_{0} + β_{1} \overset{x}{ˉ}

, which literally says the line hits the mean-point. This is a direct consequence of the residuals summing to zero.

True or false: If

β_{1} = 0

, then

x

and

y

are independent.

False.

β_{1} = 0

only means there is no linear trend. A U-shaped relationship (

y

high at both extremes of

x

) can have zero slope yet be strongly, deterministically related — the line just cannot see curved structure.

True or false: A higher

R^{2}

always means a better model.

False.

R^{2}

measures variance explained on the data you fit, not predictive skill. Adding noise features or overfitting can inflate it while generalization worsens — see 4.2-Overfitting-vs-Underfitting.

True or false:

R^{2}

can be negative.

True, but not for the OLS line on its training data. If you evaluate some other line (or a model on new data) that predicts worse than the flat line

\overset{y}{^} = \overset{y}{ˉ}

, then RSS exceeds TSS and

R^{2} = 1 - RSS / TSS

drops below zero.

True or false: The residuals from an OLS fit always add up to zero.

True (when the model has an intercept). Setting

\partial RSS / \partial β_{0} = 0

is exactly the equation

\sum (y_{i} - \overset{y}{^}_{i}) = 0

. Without an intercept term this guarantee disappears.

True or false: Swapping the roles of

x

and

y

gives a line with slope

1/ β_{1}

False. Regressing

x

y

minimizes horizontal squared errors, giving slope

s_{x y} / s_{y}^{2}

, not

s_{x}^{2} / s_{x y}

. The two lines only coincide when the fit is perfect (

R^{2} = 1

True or false: RMSE and

R^{2}

always rank two models the same way.

True for models compared on the same dataset, since

R^{2} = 1 - RSS / TSS

and RMSE

= RSS / n

both move monotonically with RSS. Across different datasets they can disagree because TSS and

n

change.

True or false: If all data points lie exactly on a horizontal line,

β_{1} = 0

and

R^{2} = 1

False on the

R^{2}

part. Here

y

has zero variance, so TSS

= 0

and

R^{2} = 1 - 0/0

is undefined. The slope is genuinely

0

, but "fraction of variance explained" is meaningless when there is no variance.

True or false: The error term

ϵ

and the residual

e_{i}

are the same thing.

False.

ϵ

is the true, unobservable noise around the real relationship;

e_{i} = y_{i} - \overset{y}{^}_{i}

is our estimated leftover using the fitted line. We only ever see residuals and hope they behave like the errors.

Spot the error

"Intercept

β_{0} = 50

means a 0-square-foot house is worth $50k, so land alone costs $50k."

The number is real but the meaning is invented.

x = 0

is far outside the observed range (extrapolation), where linearity need not hold;

β_{0}

is a mathematical anchor, not a validated price.

"My slope came out negative, so I must have made a sign mistake — slopes are positive."

Nothing is wrong. A negative

β_{1}

simply means

y

decreases as

x

increases (e.g. price vs mileage). The sign of

β_{1}

matches the sign of

Cov (x, y)

, which can legitimately be negative.

"Ice-cream sales predict drownings with

R^{2} = 0.9

, so banning ice cream will save lives."

The error is reading causation into correlation. Summer heat is a confounding variable driving both; the regression captures association only. See 2.5-Assumptions-and-Diagnostics.

"I dropped the point with the biggest residual because the model fits better without it."

A large residual is not automatically an "error" — it may be a real, informative observation. Deleting points to improve fit is data-massaging; investigate why it is far off before removing anything.

R^{2} = 0.8

, so 80% of my predictions are correct."

R^{2}

is variance explained, not an accuracy percentage. It says the model accounts for 80% of the spread in

y

, which is not a count of right-vs-wrong predictions.

"I added the residuals, got zero, and concluded my model is perfect."

Residuals summing to zero is guaranteed for any intercept model, good or terrible. It tells you nothing about fit quality; you need RSS, RMSE, or a residual plot instead.

β_{1} = Cov (x, y) / Var (y)

from the covariance formula."

Wrong denominator. The slope divides by

Var (x)

, i.e.

β_{1} = s_{x y} / s_{x}^{2}

. Using

Var (y)

would give the slope of the reversed regression.

Why questions

Why do we square residuals instead of taking absolute values?

Squaring makes RSS smooth and differentiable, so calculus gives a clean closed-form solution; absolute value has a non-differentiable kink at zero. Squaring also penalizes large errors more heavily, pulling the line away from big misses.

Why does setting the derivative to zero give the minimum and not a maximum?

RSS is a sum of squared terms, so as

β_{0}, β_{1} \to \pm \infty

it grows without bound — an upward-opening bowl (parabolic in each parameter). A single flat point on such a shape must be its lowest point.

Why must we subtract the means before computing the slope (the

(x_{i} - \overset{x}{ˉ})

terms)?

Centering isolates how

x

and

y

co-vary around their typical values, stripping out the constant offset that the intercept already handles. This turns the messy normal equations into the clean covariance-over-variance form.

Why is RMSE often preferred over RSS when reporting error?

RSS grows with sample size and is in squared units, so it is hard to interpret. RMSE divides by

n

and takes a square root, landing back in the original units of

y

("off by about ±RMSE on average").

Why does the homoscedasticity assumption matter if the OLS formula ignores it?

The formula still computes a line, but equal error variance is what makes OLS the best unbiased estimator and makes standard errors and confidence intervals trustworthy. Violate it and the point estimate survives but your uncertainty claims break.

Why can two people fit lines to the same points and, both being "correct," disagree on the slope?

If one regresses

y

x

and the other

x

y

, they minimize different distances (vertical vs horizontal). Both are valid OLS fits of different questions and only agree when the correlation is perfect.

Why does gradient descent even exist if OLS has a closed-form answer?

For one feature the closed form is trivial, but with many features (see 2.2.02-Multiple-Linear-Regression) the matrix inverse becomes expensive or unstable. 2.3-Gradient-Descent-for-Linear-Regression scales far better, which is why neural nets and 3.1-Logistic-Regression-Model rely on it.

Edge cases

What happens to the slope formula if every

x_{i}

is identical?

The denominator

\sum (x_{i} - \overset{x}{ˉ})^{2} = 0

, so

β_{1}

is undefined (division by zero). Geometrically all points sit on a vertical line, which no function

y = β_{0} + β_{1} x

can represent.

What is

R^{2}

when the model is the flat line

\overset{y}{^} = \overset{y}{ˉ}

(a "do-nothing" predictor)?

RSS then equals TSS, so

R^{2} = 1 - 1 = 0

. This is the baseline:

R^{2} = 0

means "no better than just guessing the mean."

With only two data points, what does OLS produce?

A line passing exactly through both points, giving RSS

= 0

and

R^{2} = 1

. Perfect fit here is meaningless — two points always determine a line, so there is no evidence the relationship is real.

If all

y_{i}

are the same but the

x_{i}

vary, what are

β_{0}

and

β_{1}

Cov (x, y) = 0

gives

β_{1} = 0

, and

β_{0} = \overset{y}{ˉ}

, so the fit is a flat line at that constant. But

R^{2}

is undefined since TSS

= 0

(no variance in

y

to explain).

For a single data point, is the model defined?

No. With

n = 1

\overset{x}{ˉ} = x_{1}

so the slope denominator is zero, and infinitely many lines pass through one point. You cannot estimate a trend from one observation.

Does OLS behave differently for a negative slope than a positive one?

No — the identical formula handles both; the sign simply follows

Cov (x, y)

. There is no separate "case," which is why we never branch on the sign the way angle formulas branch on quadrants.

Recall Fastest possible summary

The line always hits $(\overset{x}{ˉ}, \overset{y}{ˉ})$ ; residuals sum to zero for free; $R^{2}$ is variance explained (undefined when $y$ has no variance, negative for bad non-OLS predictors); slope is undefined when $x$ has no spread; and "the line fits" never proves " $x$ causes $y$ ."