Intuition The one core ideaMultiple regression takes a cloud of data points and slides a flat surface through them so the up-and-down gaps to the surface are as tiny as possible. Everything else — matrices, transposes, inverses — is just bookkeeping that lets a computer find that best surface for many input variables at once.
This page assumes you have seen nothing . We name every symbol the parent note Multiple regression uses, draw the picture behind it, and say why the topic cannot live without it. Read top to bottom; each block only uses symbols already built above it.
Before any formula, we need to talk about one observation and how we count many of them.
Definition Observation and the indexi observation is one complete row of measurements: one plant, one student, one house. We line them up and give each a number. The little symbol i counter that says "which row am I talking about". If we have n i 1 , 2 , 3 , … , n
n total number of observations (how many dots in the cloud).y i i x ij j i two subscripts: first says which row , second says which input .
Why we need it: regression compares a prediction to a truth for every single row, then adds up the mistakes. Without a counter we could not say "the mistake on row 3".
Worked example Reading figure s01On the left is a raw data table: each row is one observation, tagged by i right the same rows become dots in a picture, with the horizontal axis = predictor x 1 y i = 2 x 2 , 1 = 2 y 2 = 65 The whole point: a table of numbers and a cloud of dots are the same thing — regression works on the cloud.
y x 1 , … , x p
The response y
A predictor x j y
p number of predictors (how many input columns).
These are just the measured numbers — nothing is chosen or fitted yet. Next we will combine them, but to do that we first need to introduce the numbers we are allowed to tune .
β β j number we get to choose to make the fit good.
β 0 intercept : the predicted y every input is zero — where the surface crosses the vertical axis.β j j ≥ 1 slope in the direction of input x j y x j
Now that we have both the fixed inputs x j β j
Definition What "linear" means (the crucial word)A linear combination of predictors is: multiply each predictor x j β j β 0 prediction = β 0 + β 1 x 1 + β 2 x 2 + ⋯ + β p x p no bending : no x 2 x x 1 x 2 flat surface, not a curved one.
Why the topic needs it: flatness is what makes the maths solvable in one clean formula. A curved fit would have no simple closed-form answer.
ε ε i error : the part of y i not explain — measurement noise, luck, hidden causes. We assume it averages to zero:
E [ ε i ] = 0 E [ ⋅ ] average value of ". Saying the average error is zero means the surface is not systematically too high or too low.
Why both matter: the β solve for ; the ε try to make small . The whole game is choosing β ε
The error ε i can measure once we pick β residual .
Definition Residual and squared residualThe residual is the vertical gap between the real point and our surface:
e i = y i − y ^ i y ^ i prediction for row i ^ square each gap, e i 2 S = ∑ i = 1 n e i 2 ∑ i = 1 n add up , letting i n
Intuition Why square, not just take the gap?
A raw gap can be negative (point below surface); positives and negatives would cancel and hide bad fits. Squaring makes every gap positive.
Squaring punishes one big miss more than several small ones, so the surface avoids wild errors.
Squares are smooth — you can slide down them to the bottom with calculus. Absolute values have a sharp corner and are harder to solve.
Worked example Reading figure s02The black line is our fitted surface (in 1D it is just a line). The black dots are the real data; the red vertical segments joining each dot to the line are the residuals e i = y i − y ^ i vertical — regression measures the gap in the y , not the shortest slanted distance. Least squares slides the line until the total of the squared red lengths is smallest. This is the heart of Ordinary Least Squares .
Writing n
A vector is a single column of numbers. y y = y 1 y 2 ⋮ y n
Definition The parameter vectorβ ε parameter vector , and the leftover wiggles into another, the error vector :
β = β 0 β 1 ⋮ β p ( length p + 1 ) , ε = ε 1 ε 2 ⋮ ε n ( length n ) . β all the coefficients at once and bold ε all the errors at once — same numbers as before, just gathered into columns.
Definition Matrix and the design matrixX matrix is a rectangle of numbers (rows × columns). The design matrix X leading column of 1's :
X = 1 1 ⋮ 1 x 11 x 21 x n 1 ⋯ ⋯ ⋯ x 1 p x 2 p ⋮ x n p n p + 1
Intuition Why the column of 1's?Each 1 multiplies the intercept β 0 β 0 every row's prediction. Without it, the surface is forced to pass through the origin (predict 0 when all inputs are 0) — almost always wrong.
With y X β ε n y = X β + ε
Three operations power the whole derivation. We define each with its picture.
Definition Matrix–vector productX β i i X β ( X β ) i = 1 ⋅ β 0 + x i 1 β 1 + ⋯ + x i p β p = y ^ i X β all the predictions at once . That is why we bothered to stack.
X ⊤ transpose flips a matrix over its diagonal: rows become columns. If X n × ( p + 1 ) X ⊤ ( p + 1 ) × n Why it appears: to combine columns of X X ⊤ X y
∥ ⋅ ∥ v squared length is
∥ v ∥ 2 = v ⊤ v = v 1 2 + v 2 2 + ⋯ + v n 2 n ∥ y − X β ∥ 2 the squared length of the residual vector . Minimising the error is shortening an arrow.
Worked example Reading figure s03The shaded flat sheet is the column space of X X β y out of the sheet. The red arrow is the residual y − X β perpendicular to the sheet — that perpendicular drop is the best fit, and the seed of Orthogonal Projection .
Definition Dot product and "perpendicular"The dot product of two vectors a , b a ⊤ b = a 1 b 1 + ⋯ + a n b n zero , the two vectors are perpendicular (at a right angle) .
We now earn the condition that figure s03 showed geometrically. Recall S ( β ) = ∥ y − X β ∥ 2
Definition Matrix inverseFor a square matrix A inverse A − 1 A − 1 A = I I I v = v dividing .
Common mistake "Every matrix can be inverted."Why it feels right: every non-zero number can be divided by.
Fix: a matrix has no inverse when its columns are redundant (one is a combination of the others). For X ⊤ X Multicollinearity & VIF . See Matrix Inverse for the full story.
The last symbols measure how good the fit is.
y ˉ y ˉ average of all responses: y ˉ = n 1 ∑ i y i dumbest possible predictor that ignores every input.
Definition Three sums of squares
S S T = ∑ i ( y i − y ˉ ) 2 total spread of y S S E = ∑ i ( y i − y ^ i ) 2 error left after fitting (what the surface missed).S S R = ∑ i ( y ^ i − y ˉ ) 2 regression / explained part (what the surface captured).
For the least-squares fit they obey S S T = S S R + S S E
The ratio R 2 = S S R / S S T Coefficient of Determination .
Observation index i and n
Response y and predictors x
Beta dials and linear combination
Residual and sum of squares S
Vectors beta and epsilon and design matrix X
Matrix product transpose norm
Dot product and perpendicular
Mean and sums of squares SST SSE SSR
Multiple Regression estimator
Every arrow means "you must understand the source box before the target makes sense." The final box is the parent topic Multiple regression , which also opens the door to Gauss–Markov Theorem and Positive Semidefinite Matrices .
Read the left side, answer out loud, then reveal.
What does the subscript i y i Which observation (row) we mean; it runs 1 n
What does "linear" force the fitted surface to be? Flat — no curves, no squared or product terms of the inputs.
What is a residual e i The vertical gap y i − y ^ i
Why do we square the residuals before adding? To stop positives and negatives cancelling, punish big misses, and keep the total smooth for calculus.
What does the leading column of 1's in X Multiplies the intercept β 0
What is the parameter vector β The column stacking all coefficients β 0 , … , β p p + 1
What does the transpose X ⊤ Flips rows into columns so shapes line up for multiplication.
When are two vectors perpendicular in terms of the dot product? When their dot product a ⊤ b
What condition do the normal equations state geometrically? The residual is perpendicular to every column of X X ⊤ ( y − X β ^ ) = 0
Why does left-multiplying by ( X ⊤ X ) − 1 β ^ Because ( X ⊤ X ) − 1 ( X ⊤ X ) = I I β ^ = β ^
When does X ⊤ X When predictor columns are redundant (perfect collinearity).
State the identity linking the three sums of squares (least-squares fit). S S T = S S R + S S E
