Foundations — Explained variance and choosing components

Before we can talk about "explained variance," we must earn every symbol the parent note throws at you. We go from a picture of raw data all the way to the eigenvalue ratio ${\lambda }_i/{\sum }_j{\lambda }_j$ — never using a symbol before it has a plain meaning and a picture.

[!definition] Symbol 1 — A data point and a data cloud

A data point is one measured thing described by several numbers. Measure a person's height and weight and you get two numbers: that pair is one point in a 2D plane.

Stack many such points and you get a data cloud: a scatter of dots. The parent topic is entirely about the shape of this cloud.

[!definition] Symbol 2 — $n$, $d$, and the matrix $\mathbf{X}$

• $n$ = the number of data points (how many dots).
• $d$ = the number of measurements per point (the number of axes / dimensions).

We line the points up into a grid of numbers called $\mathbf{X}$: one row per point, one column per measurement. So $\mathbf{X}$ is an $n\times d$ grid ("$n$ tall, $d$ wide").

Why the topic needs it: everything else — averages, spread, covariance — is computed by sweeping over these rows and columns.

[!definition] Symbol 3 — The mean ${\bar{x}}_j$ and "centering"

The mean of column $j$, written ${\bar{x}}_j$ (say "x-bar-j"), is the average of that one measurement across all $n$ points:

${\bar{x}}_j=\frac{1}{n}{\sum }_{i=1}^n{x}_{ij}$

Here the symbol ${\sum }_{i=1}^n$ is just shorthand for "add up as $i$ counts from 1 to $n$." It is a compact "add all of these" instruction, nothing more.

Centering means subtracting each column's mean from that column, so every measurement now describes how far from average a point is. From now on, whenever we write $\mathbf{X}$ in a variance or covariance formula, we assume $\mathbf{X}$ has already been centered — every column mean is $0$.

[!definition] Symbol 4 — Variance $\text{Var}(X_j)$

Variance measures how spread out one measurement is. For centered data (mean already zero) it is the average of the squared distances from the middle:

$\text{Var}(X_j)=\frac{1}{n}{\sum }_{i=1}^n{x}_{ij}^2$

• We square so that being below average (a negative gap) counts the same as being above (a positive gap) — squares are never negative.
• Big variance = the dots are smeared far along that axis. Zero variance = every point has the same value there (a flat, dead direction).

[!definition] Symbol 5 — Total variance and the trace

Total variance is the sum of the variances of every measurement — one number describing the whole cloud's total spread:

$\text{Total Var}={\sum }_{j=1}^d\text{Var}(X_j)$

There is a name for "add up the diagonal of a square grid": the trace. For any $d\times d$ matrix $\mathbf{M}$,

$\text{trace}(\mathbf{M})={\sum }_{j=1}^d{M}_{jj}=M_{11}+M_{22}+\cdots +M_{dd},$

i.e. sum the entries running top-left to bottom-right. We meet trace here because — as we will see in Symbol 7 — the diagonal of the covariance matrix holds exactly the per-axis variances, so total variance = trace of the covariance matrix. Think of total variance as the full pie of spread that we will later slice into pieces.

[!definition] Symbol 6 — Vectors and projection $\mathbf{Xv}$

A vector $\mathbf{v}$ is an arrow pointing in some direction. When we make its length exactly 1 we call it a unit vector, written with the property ${\mathbf{v}}^T\mathbf{v}=1$ (more on the ${}^T$ below — it just means "the arrow dotted with itself gives length-squared = 1").

Projection answers: "If I shine a light straight down onto the line pointing along $\mathbf{v}$, where does each dot land?" Each landing position is a single number: how far along $\mathbf{v}$ that point sits. Collecting all these numbers for every point is written $\mathbf{Xv}$.

The transpose ${\mathbf{A}}^T$ simply means "flip the grid over its diagonal" — rows become columns. We need it so that quantities like $(\mathbf{Xv}{)}^T(\mathbf{Xv})$ read as "multiply the shadow-list by itself," which is exactly how you get a sum of squares — i.e. a variance.

[!definition] Symbol 7 — The covariance matrix $\mathbf{C}$

The covariance matrix $\mathbf{C}$ is the $d\times d$ grid built from the centered data:

$\mathbf{C}=\frac{1}{n}{\mathbf{X}}^T\mathbf{X}$

Let us open up one entry to see what it means. The matrix product ${\mathbf{X}}^T\mathbf{X}$ has, in row $j$ and column $k$, the dot product of column $j$ of $\mathbf{X}$ with column $k$. So:

$C_{jk}=\frac{1}{n}{\sum }_{i=1}^n{x}_{ij}{x}_{ik}.$

Read that entry aloud: "average, over all points, of measurement $j$ times measurement $k$." Because $\mathbf{X}$ is centered, this is exactly the covariance of measurements $j$ and $k$ — how they vary together. (If $\mathbf{X}$ were not centered, this sum would be a raw cross-product, not a covariance — which is why centering was mandatory.)

• Diagonal ($j=k$): $C_{jj}=\frac{1}{n}{\sum }_i{x}_{ij}^2=\text{Var}(X_j)$ — the per-axis variances. So $\text{trace}(\mathbf{C})={\sum }_j\text{Var}(X_j)=\text{Total Var}$, confirming Symbol 5.
• Off-diagonal ($j\ne k$): how two measurements move together (do height and weight rise as a pair?).

Why the topic needs it: the whole PCA story lives inside $\mathbf{C}$. The stretch directions of the cloud and the amount of stretch are hidden in this one grid — and eigenvalues pull them out.

[!definition] Symbol 8 — Eigenvectors ${\mathbf{v}}_i$ and eigenvalues ${\lambda }_i$

For the matrix $\mathbf{C}$, an eigenvector is a special direction that $\mathbf{C}$ does not rotate — it only stretches it. The stretch factor is its eigenvalue, written ${\lambda }_i$ ("lambda-i"). See Eigenvalues and Eigenvectors.

$\mathbf{C}{\mathbf{v}}_i={\lambda }_i{\mathbf{v}}_i$

Read this aloud: "applying $\mathbf{C}$ to the special arrow ${\mathbf{v}}_i$ gives back the same arrow, just scaled by ${\lambda }_i$."

[!formula] Symbol 8b — WHY the eigenvalue is the variance along ${\mathbf{v}}_i$

We claimed ${\lambda }_i$ equals the variance of the data projected onto ${\mathbf{v}}_i$. Let us earn it, one step at a time.

Step 1 — write the variance of the shadows. The projected data is the list $\mathbf{X}{\mathbf{v}}_i$ (Symbol 6). Its variance, since it is already centered, is the average of its squared entries, which the transpose packs neatly as:

$\text{Var}(\mathbf{X}{\mathbf{v}}_i)=\frac{1}{n}(\mathbf{X}{\mathbf{v}}_i{)}^T(\mathbf{X}{\mathbf{v}}_i).$

What this looks like: take the shadow positions in the s03 figure, square them, average them.

Step 2 — regroup the transpose. A rule of transposes: $(\mathbf{AB}{)}^T={\mathbf{B}}^T{\mathbf{A}}^T$. Applying it to $(\mathbf{X}{\mathbf{v}}_i{)}^T={\mathbf{v}}_i^T{\mathbf{X}}^T$:

$\text{Var}(\mathbf{X}{\mathbf{v}}_i)=\frac{1}{n}{\mathbf{v}}_i^T{\mathbf{X}}^T\mathbf{X}{\mathbf{v}}_i.$

Step 3 — spot the covariance matrix. But $\frac{1}{n}{\mathbf{X}}^T\mathbf{X}$ is exactly $\mathbf{C}$ (Symbol 7). Pulling the $\frac{1}{n}$ inside:

$\text{Var}(\mathbf{X}{\mathbf{v}}_i)={\mathbf{v}}_i^T\left(\frac{1}{n}{\mathbf{X}}^T\mathbf{X}\right){\mathbf{v}}_i={\mathbf{v}}_i^T\mathbf{C}{\mathbf{v}}_i.$

Step 4 — use the eigenvector property. Since $\mathbf{C}{\mathbf{v}}_i={\lambda }_i{\mathbf{v}}_i$:

${\mathbf{v}}_i^T\mathbf{C}{\mathbf{v}}_i={\mathbf{v}}_i^T({\lambda }_i{\mathbf{v}}_i)={\lambda }_i{\mathbf{v}}_i^T{\mathbf{v}}_i={\lambda }_i\cdot 1={\lambda }_i,$

using ${\mathbf{v}}_i^T{\mathbf{v}}_i=1$ from the orthonormal box. Done: $\boxed{\text{Var}(\mathbf{X}{\mathbf{v}}_i)={\lambda }_i}$. The eigenvalue is the variance along its direction — no metaphor, an equation.

Symbol 9 — Assembling the explained-variance ratio

Now every symbol in the parent formula is earned:

It is literally "how big is my slice compared to the whole pie." Adding the first $k$ slices gives the cumulative explained variance — the fraction of the cloud's spread you keep if you drop everything past component $k$. That is the number you threshold at 95% when doing Dimensionality Reduction.

The same eigenvalues also appear (as squared singular values) inside Singular Value Decomposition (SVD), and the "which columns matter" question links to Feature Selection vs. Feature Extraction. The visual ranking of ${\lambda }_i$ is drawn in a Scree Plot.

Prerequisite map

Read top to bottom: raw dots become a matrix, centering fixes the middle, variance measures spread, projection and covariance package the shape, eigenvalues slice the spread, and the ratio decides how many components to keep.

[!recall]- Quick checks as you build

Cloze yourself before moving on.

Equipment checklist

Test your readiness — cover the right side and answer.