Exercises — PCA via eigendecomposition and SVD

What we notice: $\mathbf{C}$ is already diagonal. A diagonal matrix has no coupling between features — feature 1 and feature 2 do not co-vary (off-diagonals are $0$).

Why that settles it: for a diagonal matrix the eigenvectors are just the coordinate axes ${\mathbf{e}}_1=[1,0{]}^T$ and ${\mathbf{e}}_2=[0,1{]}^T$, and the eigenvalues are the diagonal entries themselves (check: $\mathbf{C}{\mathbf{e}}_1=[4,0{]}^T=4{\mathbf{e}}_1$).

Answer:

• PC1 = $[1,0{]}^T$ with variance ${\lambda }_1=4$.
• PC2 = $[0,1{]}^T$ with variance ${\lambda }_2=1$.

False. Variance is measured from the mean; if you do not center, $\frac{1}{n-1}{\mathbf{X}}^T\mathbf{X}$ measures spread from the origin instead. The first "component" then tends to point from the origin toward the data cloud's mean (a direction that carries no information about the shape of the cloud).

Check centering: column sums are $(-1+0+1)=0$ and $(-1+0+1)=0$. Good, means are $0$.

Compute ${\mathbf{X}}_{\text{centered}}^T{\mathbf{X}}_{\text{centered}}$ (a $2\times 2$ result): ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}(-1{)}^2+0+1^2& (-1)(-1)+0+1\cdot 1\\ \text{same}& (-1{)}^2+0+1^2\end{array}\right]=\left[\begin{array}{cc}2& 2\\ 2& 2\end{array}\right].$ Divide by $n-1=2$: $\mathbf{C}=\frac{1}{2}\left[\begin{array}{cc}2& 2\\ 2& 2\end{array}\right]=\left[\begin{array}{cc}1& 1\\ 1& 1\end{array}\right].$ The off-diagonal $=1$ (equal to the diagonal) tells us the two features move together perfectly — the data lies on a line. The figure below plots these three points: they sit exactly on the $45^{\circ }$ line, which is why the second variance direction will collapse to zero.

Set up the characteristic equation $\det (\mathbf{C}-\lambda \mathbf{I})=0$: $(1-\lambda {)}^2-1=0\Rightarrow {\lambda }^2-2\lambda =0\Rightarrow \lambda (\lambda -2)=0.$ So ${\lambda }_1=2$, ${\lambda }_2=0$.

Interpret ${\lambda }_2=0$: a direction with zero variance — the data does not spread there at all (it sits exactly on a line). This is a degenerate/rank-1 case, and it is normal, not an error.

Eigenvector for ${\lambda }_1=2$: solve $(\mathbf{C}-2\mathbf{I})\mathbf{v}=0$: $\left[\begin{array}{cc}-1& 1\\ 1& -1\end{array}\right]\left[\begin{array}{c}{v}_1\\ v_2\end{array}\right]=0\Rightarrow -v_1+v_2=0\Rightarrow v_1=v_2.$ Pick ${\mathbf{v}}_1=\frac{1}{\sqrt{2}}[1,1{]}^T$ (normalized to unit length). It points along the $45^{\circ }$ line — exactly where all three points sit (see the figure in Exercise 3: PC1 runs down that line, PC2 perpendicular to it captures nothing).

Projection is a dot product $\mathbf{Z}={\mathbf{X}}_{\text{centered}}{\mathbf{v}}_1$; each row's projected value is (row)·${\mathbf{v}}_1$:

• Row $[-1,-1]$: $\frac{1}{\sqrt{2}}(-1-1)=-\frac{2}{\sqrt{2}}=-\sqrt{2}\approx -1.414$.
• Row $[0,0]$: $0$.
• Row $[1,1]$: $+\sqrt{2}\approx 1.414$.

Answer: $\mathbf{Z}=[-\sqrt{2},0,\sqrt{2}{]}^T$. One number now captures each point with zero loss (because ${\lambda }_2=0$). Geometrically (Exercise 3 figure) these are just the signed distances along the $45^{\circ }$ line from the center.

Trace = sum of eigenvalues (a theorem: trace is basis-independent and equals $\sum {\lambda }_i$): $6+3+1=10$.

Kept by PC1 (using the EVR defined at the top of the page): ${\text{EVR}}_1=\frac{6}{10}=0.6$.

Lost: $1-0.6=0.4$, i.e. $40\%$ of the total variance is discarded. Keeping one PC here is aggressive — you throw away nearly half the spread.

The key property: the covariance matrix is symmetric, and eigenvectors of a symmetric matrix belonging to different eigenvalues are orthogonal (perpendicular). So PC2 must be perpendicular to PC1.

Test the claim with a dot product (dot product $=0$ ⟺ perpendicular): $[0.8,0.6]\cdot [0.6,0.8]=0.48+0.48=0.96\ne 0.$ Not perpendicular — the claim is wrong. In the figure below, read the text labels: the arrow labelled "claimed [0.6, 0.8]" is nowhere near a right angle from the arrow labelled "PC1", whereas the arrow labelled "correct PC2" meets PC1 at a square corner.

Correct PC2: rotate PC1 by $90^{\circ }$, which swaps components and flips one sign: ${\mathbf{v}}_2=[-0.6,0.8{]}^T(\text{or its negative }[0.6,-0.8{]}^T).$ Check: $[0.8,0.6]\cdot [-0.6,0.8]=-0.48+0.48=0$. ✓ (A PC's sign is arbitrary — both $\pm \mathbf{v}$ are valid.)

Use the SVD–eigenvalue bridge ${\lambda }_i=\frac{{\sigma }_i^2}{n-1}$ (a singular value is a length, so squaring it turns it into a variance-like quantity, and dividing by $n-1$ matches the covariance's Bessel factor): ${\lambda }_1=\frac{6^2}{3}=\frac{36}{3}=12,{\lambda }_2=\frac{2^2}{3}=\frac{4}{3}\approx \mathrm{1.333.}$ EVR of PC1 (definition at the top of the page): ${\text{EVR}}_1=\frac{{\lambda }_1}{{\lambda }_1+{\lambda }_2}=\frac{12}{12+4/3}=\frac{12}{40/3}=\frac{36}{40}=\mathrm{0.9.}$ PC1 explains $90\%$ of the variance.

Center: column means are $\mu =[4,0]$. ${\mathbf{X}}_{\text{centered}}=\left[\begin{array}{cc}-2& 0\\ 0& 0\\ 2& 0\end{array}\right].$ Covariance ($n-1=2$): ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}(-2{)}^2+0+2^2& 0\\ 0& 0\end{array}\right]=\left[\begin{array}{cc}8& 0\\ 0& 0\end{array}\right]$, so $\mathbf{C}=\frac{1}{2}\left[\begin{array}{cc}8& 0\\ 0& 0\end{array}\right]=\left[\begin{array}{cc}4& 0\\ 0& 0\end{array}\right].$ Eigenvalues: diagonal ⇒ ${\lambda }_1=4,{\lambda }_2=0$. PC1: ${\mathbf{v}}_1=[1,0{]}^T$ (the axis of the nonzero variance). Project: $\mathbf{Z}={\mathbf{X}}_{\text{centered}}{\mathbf{v}}_1=[-2,0,2{]}^T$. Comment on ${\lambda }_2=0$: feature 2 is constant ($0$ variance) — a degenerate direction carrying no information. PCA correctly assigns it eigenvalue $0$, and 1D captures $100\%$ of variance (${\text{EVR}}_1=4/4=1$). This is exactly the "drop useless dimensions" use-case.

Intuition first (look at the figure below). Picture the total variance as a fixed "budget" of spread. The eigenvectors are the special axes that pile as much of that budget as possible into the first axis, then the most of what remains into the second, and so on. Any other orthonormal frame slices the same cloud at a tilt, so it spreads the budget more evenly — fine if you keep every axis, wasteful the moment you drop some.

The precise statement, sketched on this page. The variance captured by a single unit direction $\mathbf{w}$ is $g(\mathbf{w})=\frac{{\mathbf{w}}^T\mathbf{Cw}}{{\mathbf{w}}^T\mathbf{w}}(\text{the "Rayleigh quotient" — variance-along-}\mathbf{w}\text{ per unit length}).$ Write $\mathbf{w}$ in the eigenvector basis as $\mathbf{w}={\sum }_i{c}_i{\mathbf{v}}_i$ with ${\sum }_i{c}_i^2=1$. Because $\mathbf{C}{\mathbf{v}}_i={\lambda }_i{\mathbf{v}}_i$ and the ${\mathbf{v}}_i$ are orthonormal, ${\mathbf{w}}^T\mathbf{Cw}={\sum }_i{\lambda }_i{c}_i^2,$ which is a weighted average of the eigenvalues with weights $c_i^2$ summing to $1$. An average can never exceed its largest ingredient, so $g(\mathbf{w})\le {\lambda }_1$, with equality only when all the weight sits on ${\mathbf{v}}_1$ (i.e. $\mathbf{w}=\pm {\mathbf{v}}_1$). Stacking the next-best perpendicular direction gives ${\lambda }_2$, and so on — so for any $k$ orthonormal directions the captured variance is at most ${\lambda }_1+\cdots +{\lambda }_k$, achieved by the top-$k$ eigenvectors. (This "sum of the top $k$ eigenvalues is the best you can do" fact is the Ky-Fan result; we just proved the $k=1$ case above, and the general case stacks it.)

Conclusion: Projection A (true PCs) preserves at least as much variance as B, always: ${\sum }_{i\le k}{\lambda }_i\ge$ (variance of any other $k$ orthonormal frame).

When do they tie? When B happens to span the same subspace as the top-$k$ eigenvectors — e.g. if ${\lambda }_1={\lambda }_2$ (a degenerate eigenvalue), any orthonormal pair in that plane is equally good, so B can match A. Ties also occur trivially when $k=d$ (both keep everything).

Tool: the cyclic property of trace, $\text{trace}(\mathbf{AB})=\text{trace}(\mathbf{BA})$, and orthogonality $\mathbf{R}{\mathbf{R}}^T=\mathbf{I}$. $\text{trace}({\mathbf{R}}^T\mathbf{CR})=\text{trace}(\mathbf{CR}{\mathbf{R}}^T)\text{(cyclic)}=\text{trace}(\mathbf{CI})=\text{trace}(\mathbf{C}).■$ One-line justification: PCA is the special orthogonal rotation that makes $\mathbf{C}$ diagonal; since total variance (the trace) is unchanged by any orthogonal rotation, moving into the PC basis and keeping all $d$ components preserves $100\%$ of variance — it only reorganizes where the variance lives.

Setup: the score vector is $\mathbf{z}={\mathbf{X}}_{\text{centered}}\mathbf{v}$. Its scores already have mean $0$ (because the data is centered and projection is linear), so its sample variance is $\text{Var}(\mathbf{z})=\frac{1}{n-1}\Vert \mathbf{z}{\Vert }^2=\frac{1}{n-1}{\mathbf{z}}^T\mathbf{z}=\frac{1}{n-1}{\mathbf{v}}^T{\mathbf{X}}_{\text{centered}}^T{\mathbf{X}}_{\text{centered}}\mathbf{v}.$ Recognize the covariance: the middle chunk is $(n-1)\mathbf{C}$, so $\text{Var}(\mathbf{z})={\mathbf{v}}^T\mathbf{Cv}.$ Use the eigen-relation $\mathbf{Cv}=\lambda \mathbf{v}$ and unit length ${\mathbf{v}}^T\mathbf{v}=1$: ${\mathbf{v}}^T\mathbf{Cv}={\mathbf{v}}^T(\lambda \mathbf{v})=\lambda ({\mathbf{v}}^T\mathbf{v})=\lambda .■$ The eigenvalue is the variance the component explains — the central claim of PCA, proven. (This is the identity Exercise 10 leaned on for its $k=1$ case.)

The scalar factor: the student used the raw scatter matrix $\mathbf{S}={\mathbf{X}}_{\text{centered}}^T{\mathbf{X}}_{\text{centered}}$ instead of $\mathbf{C}=\frac{1}{n-1}\mathbf{S}$. Scaling a matrix by a positive constant scales every eigenvalue by that same constant, so the student's eigenvalues are $(n-1)$ times too large.

Correct the eigenvalues (divide by $n-1=4$): ${\lambda }_1=\frac{32}{4}=8,{\lambda }_2=\frac{4}{4}=1.$

Do the directions change? No. If $\mathbf{Sv}=32\mathbf{v}$, then $\mathbf{Cv}=\frac{1}{4}\mathbf{Sv}=\frac{32}{4}\mathbf{v}=8\mathbf{v}$ — the same eigenvector $\mathbf{v}$, just a rescaled eigenvalue. Positive scaling never rotates eigenvectors.

And the EVR is untouched too: ${\text{EVR}}_1=\frac{32}{32+4}=\frac{8}{8+1}=\frac{8}{9}\approx 0.889$ either way — the scalar $\frac{1}{n-1}$ cancels top and bottom.

Bottom line: the student's mistake corrupts the absolute variance values but not the principal directions nor the explained-variance ratios.

Eigenvalues sum to the trace ::: ${\sum }_i{\lambda }_i=\text{trace}(\mathbf{C})$ SVD to eigenvalue conversion ::: ${\lambda }_i={\sigma }_i^2/(n-1)$ Distinct-eigenvalue PCs are ::: orthogonal (dot product $=0$) A zero eigenvalue means ::: that direction has no variance (degenerate / redundant feature) EVR of the first k components ::: $({\lambda }_1+\cdots +{\lambda }_k)/({\lambda }_1+\cdots +{\lambda }_d)$