Visual walkthrough — PCA via eigendecomposition and SVD

We work with the tiniest possible example — a handful of dots on a flat page — because everything that happens in 1000 dimensions already happens in 2.

---

Step 1 — A cloud of dots, and the question we are asking

WHAT. We have some data: five points, each described by two numbers. Number one we call $x$ (say, hours studied), number two we call $y$ (say, exam score). Each point is a little arrow-tip on a flat sheet.

WHY. Before any math, we must know the question. Look at the cloud in the picture: it is not a round blob. It is stretched — long in one direction, thin in the other. PCA asks a single question:

Which single straight direction, if I had to keep only one, best captures how the data spreads out?

That "best direction" is what we will chase for the whole page.

PICTURE. The dots form a tilted sausage. The long axis of that sausage is the answer we want — but we do not know it yet, so it is drawn as a dashed guess.

---

Step 2 — Slide the cloud to the origin (centering)

WHAT. Compute the average point $\mathbf{\mu }$ — the balance point of the cloud — and subtract it from every dot. The whole cloud slides so its center sits exactly on $(0,0)$.

$\underset{\text{shifted cloud}}{\underbrace{{\mathbf{X}}_{\text{centered}}}}=\underset{\text{original}}{\underbrace{\mathbf{X}}}-\underset{\text{copy of the mean in every row}}{\underbrace{1_n{\mathbf{\mu }}^T}}$

• $\mathbf{\mu }=\frac{1}{n}{\sum }_i{\mathbf{x}}_i$ — the mean, i.e. average $x$ and average $y$.
• $1_n$ — a column of $n$ ones; multiplying it by ${\mathbf{\mu }}^T$ just stamps that same mean into all $n$ rows so the subtraction lines up.

WHY. PCA measures spread, and spread is always measured from the center. If we forgot to center, the very first "direction of most spread" would just point from the origin toward the far-away cloud — a direction about where the cloud sits, not how it is shaped. Centering deletes the "where" so only the "shape" remains.

PICTURE. Same sausage, now straddling the crosshairs. Its tilt (the thing we care about) is untouched; only its position changed.

---

Step 3 — Score a candidate direction by projection

WHAT. Pick any unit-length arrow $\mathbf{w}$ (a "candidate direction", length exactly 1). Drop each dot perpendicularly onto the line through $\mathbf{w}$. The shadow's signed distance from the origin is the projection:

$z_i={\mathbf{x}}_i\cdot \mathbf{w}=\Vert {\mathbf{x}}_i\Vert \cos \theta$

• ${\mathbf{x}}_i\cdot \mathbf{w}$ — the dot product: it multiplies the two arrows and reads off "how much of ${\mathbf{x}}_i$ lies along $\mathbf{w}$". We use the dot product precisely because that is the one operation that answers "how far along this direction?".
• $\theta$ — the angle between the dot and $\mathbf{w}$; when the dot lines up with $\mathbf{w}$, $\cos \theta =1$ and the shadow is longest.

WHY. A direction is "good" if the shadows spread out a lot — that means the direction runs along the data. If shadows all pile up near zero, the direction runs across the thin waist and is a poor summary.

PICTURE. One candidate arrow $\mathbf{w}$ (yellow) with dotted drop-lines from each dot to its shadow (red ticks) on the $\mathbf{w}$-line.

---

Step 4 — Turn "spread of shadows" into one number: variance

WHAT. Collect all the shadow values $z_1,\dots ,z_n$ and measure how spread out they are with the variance:

$\mathrm{Var}(z)=\frac{1}{n-1}{\sum }_{i=1}^n{z}_i^2$

(The $z_i$ already have mean 0 because we centered in Step 2, so no "$-\bar{z}$" is needed.)

• $z_i^2$ — square each shadow so that far-from-center shadows on either side count as "spread", never cancelling.
• $\frac{1}{n-1}$ — averaging, with Bessel's correction: we divide by $n-1$ instead of $n$ because one degree of freedom was already spent estimating the mean.

Now rewrite the sum using matrices. Since $z_i={\mathbf{x}}_i\cdot \mathbf{w}$, stacking gives $\mathbf{z}={\mathbf{X}}_{\text{centered}}\mathbf{w}$, and:

$\mathrm{Var}(z)=\frac{1}{n-1}\Vert {\mathbf{X}}_{\text{centered}}\mathbf{w}{\Vert }^2={\mathbf{w}}^T\underset{\mathbf{C}}{\underbrace{\left(\frac{1}{n-1}{\mathbf{X}}_{\text{centered}}^T{\mathbf{X}}_{\text{centered}}\right)}}\mathbf{w}={\mathbf{w}}^T\mathbf{Cw}$

WHY. This is the magic compression: the entire cloud's spread along any direction $\mathbf{w}$ is captured by one small $d\times d$ object $\mathbf{C}$, the covariance matrix. We no longer need the raw dots — only $\mathbf{C}$.

PICTURE. A number line of the five shadow values with their spread bracketed, labelled $\mathrm{Var}={\mathbf{w}}^T\mathbf{Cw}$.

---

Step 5 — The maximisation: why the winner is an eigenvector

WHAT. We want the $\mathbf{w}$ that makes ${\mathbf{w}}^T\mathbf{Cw}$ as large as possible, subject to $\Vert \mathbf{w}\Vert =1$. That constraint is handled with a Lagrange multiplier $\lambda$ — a bookkeeping number that lets us optimise while obeying the length rule:

$\mathcal{L}(\mathbf{w},\lambda )=\underset{\text{spread to maximise}}{\underbrace{{\mathbf{w}}^T\mathbf{Cw}}}-\lambda \underset{\text{penalty if length}\ne 1}{\underbrace{({\mathbf{w}}^T\mathbf{w}-1)}}$

A constrained maximum requires both partial derivatives to vanish — one for $\mathbf{w}$, one for $\lambda$.

Derivative w.r.t. $\mathbf{w}$ (the top of any hill is flat):

$\frac{\partial \mathcal{L}}{\partial \mathbf{w}}=2\mathbf{Cw}-2\lambda \mathbf{w}=0⟹\boxed{\mathbf{Cw}=\lambda \mathbf{w}}$

Derivative w.r.t. $\lambda$ (this is what forces the length rule back on):

$\frac{\partial \mathcal{L}}{\partial \lambda }=-({\mathbf{w}}^T\mathbf{w}-1)=0⟹{\mathbf{w}}^T\mathbf{w}=1$

The second equation just re-states $\Vert \mathbf{w}\Vert =1$ — it guarantees our winning direction is a genuine unit vector, closing the optimisation properly.

WHY the derivative? The best direction sits at the peak of the spread-versus-angle curve. At a peak the slope is zero — that is exactly what setting the derivative to zero finds. We use calculus here because "maximise a smooth quantity" is the one question derivatives were built to answer.

The boxed line $\mathbf{Cw}=\lambda \mathbf{w}$ is the eigenvalue equation: the best direction $\mathbf{w}$ is an eigenvector of $\mathbf{C}$. And plugging it back (using ${\mathbf{w}}^T\mathbf{w}=1$ from the second condition):

$\mathrm{Var}={\mathbf{w}}^T\mathbf{Cw}={\mathbf{w}}^T(\lambda \mathbf{w})=\lambda \Vert \mathbf{w}{\Vert }^2=\lambda .$

So the eigenvalue $\lambda$ is the variance along its eigenvector.

PICTURE. A polar plot: spread ${\mathbf{w}}^T\mathbf{Cw}$ as a function of the arrow's angle. The two peaks sit exactly where the arrow aligns with an eigenvector; the taller peak is ${\lambda }_1$.

---

Step 6 — All the directions at once, and every case covered

WHAT. $\mathbf{C}$ is symmetric, so it has $d$ real eigenvectors ${\mathbf{v}}_1,\dots ,{\mathbf{v}}_d$ that are mutually perpendicular, with eigenvalues sorted ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_d\ge 0$. These are the principal components: ${\mathbf{v}}_1$ = biggest-spread direction, ${\mathbf{v}}_2$ = biggest-spread direction among those perpendicular to ${\mathbf{v}}_1$, and so on.

WHY $\ge 0$? $\mathbf{C}$ is positive semi-definite (it is a sum of squares), so no eigenvalue can be negative — you cannot have negative spread.

Every case:

• Round blob (${\lambda }_1={\lambda }_2$): no direction wins; PCA has no preferred axis (the polar plot is a circle). PCA still runs, but the axes are arbitrary.
• Perfect line (${\lambda }_2=0$): all spread on one axis, zero on the other. The second PC captures nothing — you can drop it losslessly.
• A single point / all points identical: $\mathbf{C}=0$, all ${\lambda }_i=0$. Degenerate: there is no direction of spread at all.
• One feature on a huge scale (e.g. metres vs millimetres): that feature fakes a large $\lambda$ purely from units — this is why you scale first (2.4.03-Feature-Scalingand-Normalization).

PICTURE. The ellipse of the cloud with both eigenvectors drawn — long axis ${\mathbf{v}}_1$ (blue), short axis ${\mathbf{v}}_2$ (green), perpendicular, with the three degenerate ellipses shown as small insets.

---

Step 7 — Project, and the SVD shortcut

WHAT. To compress to $k$ dimensions, stack the top $k$ eigenvectors as columns of ${\mathbf{V}}_k$ and take shadows:

$\mathbf{Z}={\mathbf{X}}_{\text{centered}}{\mathbf{V}}_k,{\mathbf{V}}_k=[{\mathbf{v}}_1\mid \cdots \mid {\mathbf{v}}_k]$

$\mathbf{Z}$ is $n\times k$ — the same $n$ points described by only $k$ numbers each.

The SVD shortcut. We never actually needed to build $\mathbf{C}$. Any centered matrix factors as ${\mathbf{X}}_{\text{centered}}=\mathbf{U}\mathbf{\Sigma }{\mathbf{V}}^T$ (Linear-Algebra-SVD). Its right singular vectors $\mathbf{V}$ are already the principal components, and:

${\lambda }_i=\frac{{\sigma }_i^2}{n-1}$

• ${\sigma }_i$ — the $i$-th singular value (how strongly the data stretches along ${\mathbf{v}}_i$).
• Squaring and dividing by $n-1$ converts "stretch of the data" into "variance". This matches Step 5's $\lambda$ exactly.

WHY prefer SVD? Forming $\mathbf{C}$ squares the numbers and can lose precision; SVD works on ${\mathbf{X}}_{\text{centered}}$ directly — more numerically stable, and faster when features $\gg$ samples.

PICTURE. The 2D cloud collapsing onto the ${\mathbf{v}}_1$ line: dots turning into their 1D shadow values, "2 numbers → 1 number".

---

The one-picture summary

Everything above, in a single frame: raw cloud → centered → the covariance ellipse → its two eigen-axes → projection onto PC1.

Recall Feynman retelling — say it back in plain words

I had a cloud of dots that was long in one direction and thin in another. First I slid the cloud so its middle sat on the origin, because I only care about its shape, not where it is. Then, to test any direction, I dropped every dot's shadow onto a line and measured how spread out those shadows were — that "spread" is variance, and I found it equals a compact formula ${\mathbf{w}}^T\mathbf{Cw}$, where $\mathbf{C}$ is one small matrix that remembers the whole cloud. I wanted the direction with the most spread, so I climbed to the top of the spread-hill by setting both derivatives (in $\mathbf{w}$ and in $\lambda$) to zero, and out popped $\mathbf{Cw}=\lambda \mathbf{w}$ with $\mathbf{w}$ a unit vector — the best direction is an eigenvector, and its eigenvalue is its variance. The eigenvectors come out perpendicular and ranked, giving me my principal components. Finally I could either eigen-decompose $\mathbf{C}$ or, more cleanly, run SVD straight on the data, where the right singular vectors are already the answer and ${\lambda }_i={\sigma }_i^2/(n-1)$.

Recall Quick self-test

Why must we center before PCA? ::: Variance is measured from the mean; without centering PC1 just points at the cloud's location instead of its shape. What does the eigenvalue ${\lambda }_i$ physically mean? ::: The variance (spread) of the data along principal component ${\mathbf{v}}_i$. How do singular values relate to eigenvalues? ::: ${\lambda }_i={\sigma }_i^2/(n-1)$. What happens to PC2 when the cloud is a perfect line? ::: Its eigenvalue is 0 — it captures no variance and can be dropped losslessly. Does dividing by $n$ instead of $n-1$ change the principal directions? ::: No — only the eigenvalues rescale by $\frac{n-1}{n}$; the eigenvectors and variance ratios are unchanged.

Related: 2.5.01-Introduction-to-Unsupervised-Learning · 2.5.07-K-Means-Clustering · 2.6.04-Kernel-PCA · 3.2.05-Autoencoders (a nonlinear generalisation of this same compress-then-reconstruct idea).