Visual walkthrough — Explained variance and choosing components

This page rebuilds the central result of Explained variance and choosing components from absolute zero, one picture at a time. By the end you will see why one number — an eigenvalue — is the same thing as "spread along a direction", and why adding those numbers up tells you how many directions to keep.

We assume nothing. If a symbol appears, it was drawn first.

---

Step 1 — Draw the data as a centered cloud

WHAT. We start with a table of measurements. Every row is one data point (say, one person); every column is one feature (height, weight). Two columns means every point is a dot on a flat plane. We call this table $\mathbf{X}$ and give it $n$ rows and $d$ columns.

WHY. Before we can talk about "spread", every quantity must be measured from the middle of the cloud, not from an arbitrary zero. So we center: subtract each column's average so the cloud sits centered on the origin. Centering is what lets a single squared distance stand for "how far from typical".

PICTURE. The blue dots are the raw cloud; the pink dots are the same cloud slid so its center of mass lands on the crossing of the axes.

---

Step 2 — Measure spread along the axes (variance)

WHAT. Pick one column $j$. Take each point's value, square it, average the squares. That average of squared distances-from-center is the variance of that column.

$\text{Var}(X_j)=\underset{\text{average}}{\underbrace{\frac{1}{n}}}{\sum }_{i=1}^n\underset{\text{squared distance of point }i\text{ from center}}{\underbrace{(x_{ij}-{\bar{x}}_j{)}^2}}$

WHY squared, why not just distance? Squaring does two jobs at once: it kills the sign (left and right both count as spread), and it punishes far-out points more, so variance genuinely tracks width. The $\frac{1}{n}$ turns a sum into a per-point average so bigger datasets don't automatically look "more spread".

PICTURE. Each dashed segment is one point's distance from center along the horizontal axis; the shaded bar area is the average of the squares of those segments.

---

Step 3 — Spread depends on which direction you look

WHAT. Variance was measured along the given axes. But nothing forces us to look along the original axes. Choose any direction — call it a unit-length arrow $\mathbf{v}$ — and slide every dot onto that arrow (its shadow / projection). Those shadows have their own variance.

WHY. The original axes are an accident of how the data was recorded. The cloud is often widest along a tilted line. If we want the directions that carry the most spread, we must be free to rotate our "measuring stick" $\mathbf{v}$.

PICTURE. The same cloud measured two ways: along a poor direction (thin shadow spread, small variance) and along the cloud's long diagonal (fat shadow spread, large variance).

---

Step 4 — The variance of a shadow is ${\mathbf{v}}^T\mathbf{C}\mathbf{v}$

WHAT. We compute the variance of the shadow list $\mathbf{Xv}$. Because shadows are already centered (centered cloud → centered shadows), variance is average of squared shadow-positions:

$\text{Var}(\mathbf{Xv})=\frac{1}{n}(\mathbf{Xv}{)}^T(\mathbf{Xv})={\mathbf{v}}^T\underset{\text{this is }\mathbf{C}}{\underbrace{\left(\frac{1}{n}{\mathbf{X}}^T\mathbf{X}\right)}}\mathbf{v}={\mathbf{v}}^T\mathbf{C}\mathbf{v}$

WHY. $(\mathbf{Xv}{)}^T(\mathbf{Xv})$ is "sum of squared shadows" written compactly (a dot product of the shadow list with itself). Factoring out $\mathbf{v}$ from both sides reveals a single object in the middle that does not depend on $\mathbf{v}$ — the covariance matrix $\mathbf{C}$. This is the whole reason $\mathbf{C}$ exists: it is the "spread machine" that, sandwiched between any direction $\mathbf{v}$, returns the spread along that direction.

PICTURE. A box labelled $\mathbf{C}$ with $\mathbf{v}$ going in one side and the spread number coming out the other; turning the input arrow rotates the shadow and changes the output.

---

Step 5 — Eigenvectors are the special directions, eigenvalues ARE their spread

WHAT. Among all directions $\mathbf{v}$, the special ones are the eigenvectors: directions the spread-machine does not rotate. Feeding an eigenvector ${\mathbf{v}}_i$ into $\mathbf{C}$ gives back the same direction, only stretched by a number ${\lambda }_i$:

$\mathbf{C}{\mathbf{v}}_i={\lambda }_i{\mathbf{v}}_i$

Plug this into Step 4's formula:

$\text{Var}(\mathbf{X}{\mathbf{v}}_i)={\mathbf{v}}_i^T\mathbf{C}{\mathbf{v}}_i={\mathbf{v}}_i^T({\lambda }_i{\mathbf{v}}_i)={\lambda }_i\underset{=1}{\underbrace{({\mathbf{v}}_i^T{\mathbf{v}}_i)}}={\lambda }_i$

WHY this is the punchline. The spread along an eigenvector is exactly its eigenvalue — no extra computation. The last "$=1$" uses that eigenvectors are unit length. So the abstract number ${\lambda }_i$ that pops out of linear algebra is a concrete physical thing: the variance the data has along principal component $i$. (These same ${\lambda }_i$ are the squared singular values you meet in Singular Value Decomposition (SVD).)

PICTURE. Two eigenvector arrows on the cloud; each labelled with its own eigenvalue, the arrow's length drawn proportional to $\sqrt{{\lambda }_i}$ so the long axis literally is the big eigenvalue.

Recall Why does

${\mathbf{v}}_i^T{\mathbf{v}}_i=1$? Because principal components are chosen to be unit-length arrows (length one). Dotting a unit arrow with itself gives its squared length $=1$. If we forgot to normalize, the "$=1$" would become the arrow's squared length and the neat result ${\lambda }_i$ would be scaled — that is exactly the bug to avoid.

---

Step 6 — Total spread = sum of ALL eigenvalues

WHAT. Add every column's variance (the diagonal of $\mathbf{C}$) — that is the total spread from Step 2. A linear-algebra identity says the diagonal-sum (trace) equals the eigenvalue-sum:

$\text{Total Var}=\text{trace}(\mathbf{C})={\sum }_{j=1}^d{\lambda }_j$

WHY. Rotating your measuring axes to the eigenvector directions does not create or destroy spread — it only re-labels it. So the total is the same whether you sum along the original axes or along the eigenvectors. This gives us the one missing piece: a denominator.

PICTURE. A single bar of total length split into slices, one per eigenvalue — the whole bar is the total variance, each slice one ${\lambda }_i$.

---

Step 7 — Divide: explained-variance ratio and its cumulative sum

WHAT. Now the definition writes itself. The fraction of spread carried by component $i$ is its slice over the whole bar:

${\text{EVR}}_i=\frac{{\lambda }_i}{\sum _{j=1}^d{\lambda }_j}\text{cumulative: }{\text{CEV}}_k=\frac{\sum _{i=1}^k{\lambda }_i}{\sum _{j=1}^d{\lambda }_j}$

WHY division and not subtraction? A ratio is scale-free: it answers "what fraction of the information do I keep?" regardless of the data's units. ${\text{CEV}}_k$ climbs from near $0$ toward $1$ as you add components; picking $k$ is reading off where that climb crosses your target line (say $0.95$). This is the same curve you plot in a Scree Plot.

PICTURE. The staircase: eigenvalue slices on the left, the cumulative curve on the right climbing to a horizontal $95\%$ threshold, with the crossing $k$ marked.

---

Step 8 — Edge and degenerate cases

Every scenario must be covered, or the reader hits a wall you didn't build.

Case A — a zero eigenvalue (${\lambda }_i=0$). Some direction has no spread; every point has the same shadow there. That component contributes ${\text{EVR}}_i=0$ and can be dropped with zero loss. This is exactly a redundant / perfectly-correlated feature.

Case B — all eigenvalues equal (${\lambda }_1=\cdots ={\lambda }_d$). The cloud is a perfectly round ball. No direction is special, ${\text{EVR}}_i=1/d$ for all $i$, and the cumulative curve is a straight diagonal line. Here PCA gives you nothing — you must keep almost all components to hit $95\%$. That flatness is the warning sign.

Case C — one dominant eigenvalue. The cloud is a thin needle. ${\text{EVR}}_1$ is near $1$, the curve jumps to the top in one step, and a single component suffices.

PICTURE. Three little clouds (flat needle / round ball / general) each with its cumulative curve underneath — steep jump, straight diagonal, moderate knee.

---

The one-picture summary

This single figure chains the whole argument: centered cloud → measure spread along a rotating arrow → the spread-machine $\mathbf{C}$ → its no-rotate directions carry spread ${\lambda }_i$ for free → stack the ${\lambda }_i$ into a total → divide to get the cumulative curve → read off $k$ at your threshold. That is Dimensionality Reduction made into one honest number.

Recall Feynman retelling — say it like you'd tell a friend

Toss a handful of dots on a table. They make a blob that's fatter one way than another. "Variance" is just: pick a direction, look at how spread out the dots are along it — square the distances so left and right both count, average them. Now here's the trick: there's a little machine (the covariance matrix) where you push in a direction-arrow and it hands you back the spread along that arrow. Most arrows get twisted by the machine, but a few special arrows come back pointing the exact same way, only longer — those are the principal directions, and the amount they got stretched (the eigenvalue) is the spread along them. Add up all those stretch-numbers and you've got the total spread. So each direction's share is its stretch-number divided by the total. Line them up biggest-first, keep adding shares until you've collected 95% of the total, and stop — the number of arrows you grabbed is how many dimensions you keep. Flat blob? Almost every arrow matters, keep lots. Needle blob? One arrow has it all, keep one.

Recall Rebuild the chain from memory

$\text{Var along }\mathbf{v}$ equals? ::: ${\mathbf{v}}^T\mathbf{Cv}$ Var along eigenvector ${\mathbf{v}}_i$ equals? ::: ${\lambda }_i$ Total variance equals? ::: ${\sum }_j{\lambda }_j=\text{trace}(\mathbf{C})$ When is the cumulative curve a straight diagonal? ::: all eigenvalues equal (a round cloud) A zero eigenvalue means? ::: a direction with no spread — drop it for free