Worked examples — PCA via eigendecomposition and SVD

This page is the "hands dirty" companion to the parent PCA note. There we derived why eigenvectors of the covariance matrix are the directions of maximum variance. Here we run the machine on every kind of input it can meet — clean 2D data, data with zero variance in one direction, data where two directions tie, already-centered data, a real-world word problem, and an exam twist. If you skipped the parent, read its Step 1–4 first; everything below assumes you know these words:

Recall Vocabulary you must already own (click to check yourself)

centered data ::: original data with the mean subtracted, so the cloud sits at the origin covariance matrix $\mathbf{C}$ ::: a $d\times d$ symmetric table; diagonal = variances, off-diagonal = how features move together eigenvector / eigenvalue ::: a direction $\mathbf{v}$ that $\mathbf{C}$ only stretches (not rotates), and the stretch factor $\lambda$ = variance along it principal component (PC) ::: an eigenvector of $\mathbf{C}$, ordered by its eigenvalue singular value $\sigma$ ::: from SVD of the data matrix; connected by $\lambda ={\sigma }^2/(n-1)$

If any of those are fuzzy, revisit Linear-Algebra-Eigenvalues-and-Eigenvectors and Linear-Algebra-SVD.

---

A note on characteristic polynomials

Every 2D example below solves $\det (\mathbf{C}-\lambda \mathbf{I})=0$. Rather than pull the coefficients "by inspection," here is the general rule we will reuse each time.

We cite this box every time so no coefficient appears unexplained.

---

The scenario matrix

Before working anything, let us list every distinct situation a PCA problem can be. A worked example is only useful if it covers a cell of this table that the others don't.

#	Cell (the scenario)	What makes it different	Example
A	Generic tilted cloud, ${\lambda }_1\gg {\lambda }_2>0$	the "normal" case, one clear dominant direction	Ex 1
B	Axis-aligned data (off-diagonals $=0$)	PCs are just the original axes — a sanity floor	Ex 2
C	Degenerate: one direction has zero variance (${\lambda }_2=0$)	data lies on a perfect line; rank-deficient $\mathbf{C}$	Ex 3
D	Isotropic tie (${\lambda }_1={\lambda }_2$)	no unique PC — direction is ambiguous	Ex 4
E	Negative covariance (cloud tilts "down")	sign of off-diagonal flips the PC into a different quadrant	Ex 5
F	Scaling matters (unscaled vs scaled features)	why we sometimes standardise first	Ex 6
G	Real-world word problem (age vs heart-rate)	translate a story into $\mathbf{X}$, interpret the PC	Ex 7
H	Exam twist: given eigenvalues, choose $k$	reverse direction — pick dimensionality from EVR	Ex 8

We now hit each cell. Wherever geometry appears, look at the figure — the arrows are the argument.

---

Ex 1 — Cell A: the generic tilted cloud

Forecast: the points climb roughly along a line — guess PC1 points up-and-to-the-right, and captures almost all variance. Will ${\lambda }_1$ be near the total?

The figure below plots the five data points (black dots) with the mean marked by a black cross; the red arrow is PC1, drawn from the mean along the direction that eigen-analysis will produce. Notice the arrow lies inside the elongated cloud — that is what "direction of maximum variance" looks like.

1. Center. Mean $\mathbf{\mu }=[3,5.2{]}^T$, so ${\mathbf{X}}_c=\left[\begin{array}{cc}-2& -3.2\\ -1& -1.2\\ 0& -0.2\\ 1& 1.8\\ 2& 2.8\end{array}\right].$ Why this step? PCA measures spread from the mean; if we don't move the cloud to the origin, PC1 would waste itself pointing at the mean.

2. Covariance with $n-1=4$: $\mathbf{C}=\frac{1}{4}{\mathbf{X}}_c^T{\mathbf{X}}_c=\left[\begin{array}{cc}2.5& 3.75\\ 3.75& 5.7\end{array}\right].$ Why this step? We need the object whose eigenvectors are the variance directions — that object is $\mathbf{C}$.

3. Eigenvalues via the template. Here $a=2.5,b=3.75,c=5.7$, so trace $=a+c=8.2$ and $\det \mathbf{C}=ac-b^2=2.5(5.7)-3.75^2=14.25-14.0625=0.1875$. Dropping these into the boxed formula: ${\lambda }^2-8.2\lambda +0.1875=0\Rightarrow {\lambda }_{1,2}=\frac{8.2\pm \sqrt{8.2^2-4(0.1875)}}{2}.$ ${\lambda }_1\approx 8.177,{\lambda }_2\approx 0.0229$. Why this step? The eigenvalue is the variance along its direction (parent's Lagrange proof). Big ${\lambda }_1$ = long axis.

4. Eigenvector for ${\lambda }_1$. Solve $(\mathbf{C}-{\lambda }_1\mathbf{I})\mathbf{v}=0$, normalise: ${\mathbf{v}}_1\approx [0.558,0.830{]}^T$. Why this step? This unit arrow is PC1 — the red arrow in the figure.

5. Project onto PC1: $\mathbf{Z}={\mathbf{X}}_c{\mathbf{v}}_1\approx [-3.77,-1.55,-0.17,2.05,3.44{]}^T$. Why this step? Each 2D point collapses to one number = its position along the long axis. We threw away only the tiny ${\lambda }_2$.

Verify: EVR of PC1 $={\lambda }_1/({\lambda }_1+{\lambda }_2)=8.177/8.2\approx 0.9972$ — 99.7% captured. Trace check: ${\lambda }_1+{\lambda }_2=8.2=2.5+5.7=\text{trace}(\mathbf{C})$. ✓

---

Ex 2 — Cell B: axis-aligned data (sanity floor)

Forecast: if features are independent and axis-aligned, PCA should hand back the original $x$- and $y$-axes — no rotation. PC1 = whichever axis spreads more.

1. Center. $\mathbf{\mu }=[2,1{]}^T$, ${\mathbf{X}}_c=\left[\begin{array}{cc}-2& -1\\ 2& -1\\ -2& 1\\ 2& 1\end{array}\right]$. Why this step? Same reason as always — variance is measured from the mean.

2. Covariance ($n-1=3$): off-diagonal $=\frac{1}{3}\sum x_i{y}_i=\frac{1}{3}(2-2-2+2)=0$. $\mathbf{C}=\left[\begin{array}{cc}\frac{16}{3}& 0\\ 0& \frac{4}{3}\end{array}\right].$ Why this step? The zero off-diagonal is the whole point — it says the features are uncorrelated.

3. Eigenvalues/vectors. A diagonal matrix already displays its eigenvalues (the template gives $b=0$, so ${\lambda }^2-(a+c)\lambda +ac=0$ factors as $(\lambda -a)(\lambda -c)=0$): ${\lambda }_1=16/3,{\lambda }_2=4/3$, with ${\mathbf{v}}_1=[1,0{]}^T$, ${\mathbf{v}}_2=[0,1{]}^T$. Why this step? Diagonal $\Rightarrow$ eigenvectors are the standard axes.

Verify: ${\lambda }_1+{\lambda }_2=20/3=\text{trace}$. ✓ PC1 is the $x$-axis because $x$ spreads more ($16/3>4/3$). This is the floor every PCA must respect: aligned data yields aligned PCs.

---

Ex 3 — Cell C: degenerate, zero-variance direction

Forecast: the cloud is a perfect line. There is genuinely nothing to see off that line — so ${\lambda }_2$ should be exactly $0$. PCA discovers the line is 1-dimensional.

The figure shows all four points lying exactly on a straight line; the red arrow is PC1, pointing along that line. There is no spread perpendicular to the arrow at all — that is why the second eigenvalue will be zero.

1. Center. $\mathbf{\mu }=[2.5,5{]}^T$, ${\mathbf{X}}_c=\left[\begin{array}{cc}-1.5& -3\\ -0.5& -1\\ 0.5& 1\\ 1.5& 3\end{array}\right]$.

2. Covariance ($n-1=3$): $\sum x^2=5,\sum xy=10,\sum y^2=20$. $\mathbf{C}=\frac{1}{3}\left[\begin{array}{cc}5& 10\\ 10& 20\end{array}\right].$ Why this step? Notice the second row is exactly twice the first — the matrix is rank 1. That's the algebraic fingerprint of "data on a line."

3. Eigenvalues via the template. Here $a=5/3,b=10/3,c=20/3$: trace $=a+c=25/3$ and $\det \mathbf{C}=ac-b^2=\frac{1}{9}(100-100)=0$. So ${\lambda }^2-\frac{25}{3}\lambda +0=0$, giving ${\lambda }_1=25/3\approx 8.33,{\lambda }_2=0$. Why this step? A zero eigenvalue is not an error — it's the honest statement "zero variance in that direction" (which the zero determinant already forecast).

4. PC1 solves $(\mathbf{C}-{\lambda }_{1}I)\mathbf{v}=0\Rightarrow {\mathbf{v}}_1=\frac{1}{\sqrt{5}}[1,2{]}^T\approx [0.447,0.894{]}^T$, exactly the line's direction. Why this step? PCA recovers the slope-2 line: the arrow lies on the data.

Verify: ${\lambda }_1+{\lambda }_2=25/3=\text{trace}=\frac{1}{3}(5+20)$. ✓ EVR of PC1 $=100\%$. Reducing to $k=1$ loses nothing — this is the ideal compression.

---

Ex 4 — Cell D: isotropic tie (ambiguous PCs)

Forecast: the cloud looks the same in every direction (isotropic). There should be no unique winner — any direction has equal variance.

The figure plots the four points forming a plus/diamond shape, with a red circle drawn around them: the circle signals that every radial direction is equally "long," so no single arrow can claim to be PC1.

1. Center. Mean is already $[0,0{]}^T$, so ${\mathbf{X}}_c=\mathbf{X}$. Why this step? Always confirm — here centering is a no-op, but skipping the check is where bugs hide.

2. Covariance ($n-1=3$): $\sum x^2=2,\sum y^2=2,\sum xy=0$. $\mathbf{C}=\frac{1}{3}\left[\begin{array}{cc}2& 0\\ 0& 2\end{array}\right]=\frac{2}{3}\mathbf{I}.$ Why this step? A scalar multiple of the identity is the algebraic sign of "same spread everywhere."

3. Eigenvalues via the template. Here $a=c=2/3,b=0$: ${\lambda }^2-\frac{4}{3}\lambda +\frac{4}{9}=0=(\lambda -\frac{2}{3}{)}^2$. A repeated root ${\lambda }_1={\lambda }_2=2/3$. For $c\mathbf{I}$, every vector is an eigenvector. Why this step? The tie means PC1 is not defined — $[1,0]$, $[0,1]$, $[\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}}]$ are all equally valid.

Verify: trace $=4/3=2\times \frac{2}{3}$. ✓ EVR of any single PC $=50\%$.

---

Ex 5 — Cell E: negative covariance (down-tilting cloud)

Forecast: negative correlation $\Rightarrow$ the long axis runs upper-left to lower-right. PC1's two components should have opposite signs.

The figure shows the four points sloping downward left-to-right; the red arrow is PC1, running from lower-right to upper-left. Its horizontal and vertical components have opposite signs — that opposite-sign pattern is the visible fingerprint of negative covariance.

1. Center. $\mathbf{\mu }=[1.5,1.75{]}^T$, ${\mathbf{X}}_c=\left[\begin{array}{cc}-1.5& 1.25\\ -0.5& 0.25\\ 0.5& 0.25\\ 1.5& -1.75\end{array}\right]$.

2. Covariance ($n-1=3$): $\sum x^2=5,\sum y^2=4.75,\sum xy=-4.5$. $\mathbf{C}=\frac{1}{3}\left[\begin{array}{cc}5& -4.5\\ -4.5& 4.75\end{array}\right]\approx \left[\begin{array}{cc}1.667& -1.5\\ -1.5& 1.583\end{array}\right].$ Why this step? The negative off-diagonal is the number that will drag PC1 into the second/fourth quadrant.

3. Eigenvalues via the template. With $a=5/3,b=-3/2,c=4.75/3$: trace $=a+c=3.25$ and $\det \mathbf{C}=ac-b^2=\frac{1}{9}(5\cdot 4.75-4.5^2)=\frac{1}{9}(23.75-20.25)=\frac{3.5}{9}\approx 0.389$. So $\lambda =\frac{3.25\pm \sqrt{3.25^2-4(0.389)}}{2}\Rightarrow {\lambda }_1\approx 3.125,{\lambda }_2\approx \mathrm{0.125.}$ Why this step? Same variance-finding as before; nothing about signs changes the method — only the value of $b$.

4. PC1. Solve $(\mathbf{C}-{\lambda }_{1}I)\mathbf{v}=0\Rightarrow {\mathbf{v}}_1\approx [0.718,-0.696{]}^T$ (opposite signs — points down-right). Why this step? The sign pattern is the whole lesson: negative covariance flips the arrow across the horizontal.

Verify: ${\lambda }_1+{\lambda }_2\approx 3.25=\text{trace}$. ✓ The product of PC1's components is negative, confirming the down-tilt. Compare with Ex 1 where both were positive.

---

Ex 6 — Cell F: why scaling changes the answer

Forecast: height's numbers are ~30× bigger, so raw PCA's PC1 will point almost entirely along height. Standardising should let weight compete.

1. Raw center. $\mathbf{\mu }=[1750,57.5{]}^T$. Centered height column is $\{-150,-50,50,150\}$, weight $\{-7.5,2.5,-2.5,7.5\}$.

2. Raw covariance ($n-1=3$): $\text{Var}(h)=\frac{1}{3}(22500+2500+2500+22500)=16666.7$, $\text{Var}(w)=\frac{1}{3}(56.25+6.25+6.25+56.25)=41.67$. Why this step? $16666.7\gg 41.67$ — height dominates purely because its units are larger.

3. Raw PC1 is essentially $[1,\approx 0{]}^T$: it tracks height, ignoring weight almost entirely. Why this step? Shows the danger — PCA is not scale-invariant.

4. Standardise each column to unit variance (divide by its std). Now both variances equal $1$, so $\mathbf{C}$ becomes the correlation matrix $\left[\begin{array}{cc}1& \rho \\ \rho & 1\end{array}\right]$ with $\rho =\text{corr}(h,w)$. Compute the cross term $\sum h_c{w}_c=(-150)(-7.5)+(-50)(2.5)+(50)(-2.5)+(150)(7.5)=1125-125-125+1125=2000$. Then $\rho =\frac{\sum h_c{w}_c}{\sqrt{\sum h_c^2}\sqrt{\sum w_c^2}}=\frac{2000}{\sqrt{50000}\sqrt{125}}=\frac{2000}{2500}=\mathrm{0.80.}$ Why this step? Standardising strips units so the pattern (correlation), not the magnitude, drives PCA.

5. Standardised eigenvalues via the template on $\left[\begin{array}{cc}1& 0.8\\ 0.8& 1\end{array}\right]$: trace $=2$, $\det =1-0.8^2=0.36$, so ${\lambda }^2-2\lambda +0.36=0\Rightarrow \lambda =1\pm 0.8$, giving ${\lambda }_1=1.8,{\lambda }_2=0.2$, with ${\mathbf{v}}_1=[\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}}{]}^T$. Why this step? Now PC1 splits equally between height and weight — the honest joint direction.

Verify: ${\lambda }_1+{\lambda }_2=2=\text{trace}$ (two standardised features). ✓ EVR of PC1 $=1.8/2=90\%$. See 2.4.03-Feature-Scalingand-Normalization — always standardise when units differ.

---

Ex 7 — Cell G: real-world word problem

Forecast: older members have lower heart-rate here — expect negative correlation, PC1 pointing "older + fitter." One number should summarise each member well.

1. Build $\mathbf{X}$ and center. $\mathbf{\mu }=[35,73{]}^T$. Centered ages $\{-15,-5,5,15\}$, HR $\{7,2,-1,-8\}$. Why this step? Translating the story into a matrix is the real skill — rows = people, columns = measurements.

2. Covariance ($n-1=3$): $\text{Var(age)}=\frac{1}{3}(225+25+25+225)=166.67$; $\text{Var(HR)}=\frac{1}{3}(49+4+1+64)=39.33$; $\text{Cov}=\frac{1}{3}((-15)(7)+(-5)(2)+(5)(-1)+(15)(-8))=\frac{1}{3}(-105-10-5-120)=-80$. $\mathbf{C}=\left[\begin{array}{cc}166.67& -80\\ -80& 39.33\end{array}\right].$ Why this step? Negative covariance confirms our forecast: age up, HR down.

3. Eigenvalues via the template. $a=166.67,b=-80,c=39.33$: trace $=a+c=206$ and $\det \mathbf{C}=ac-b^2=166.67\cdot 39.33-(-80{)}^2=6556-6400=156$. So $\lambda =\frac{206\pm \sqrt{206^2-4\cdot 156}}{2}\Rightarrow {\lambda }_1\approx 205.24,{\lambda }_2\approx \mathrm{0.76.}$

4. PC1 $\approx [0.899,-0.438{]}^T$. Why this step? Interpret: moving along PC1 means age increases while heart-rate decreases — a single "aging-and-getting-fitter-here" axis.

Verify: ${\lambda }_1+{\lambda }_2\approx 206=\text{trace}$. ✓ EVR $=205.24/206\approx 99.6\%$ — one number describes each member almost perfectly.

---

Ex 8 — Cell H: exam twist — choose $k$ from eigenvalues

Forecast: most variance is in the first two or three — guess $k=3$.

1. Total variance $=12+5+2+0.6+0.3+0.1=20$. Why this step? $\text{EVR}$ is a fraction of the total, so we need the denominator first.

2. Cumulative EVR:

$k$	cumulative $\sum \lambda$	EVR
1	12	0.600
2	17	0.850
3	19	0.950
4	19.6	0.980

Why this step? We scan until the running fraction first reaches the $0.95$ threshold.

3. Decide: at $k=3$, EVR $=19/20=0.95$ exactly — the first $k$ meeting the bar. Choose $k=3$. Why this step? The rule "keep 95% of variance" (parent's Step-choosing section) picks the elbow automatically.

Verify: ${\text{EVR}}_3=19/20=0.95\ge 0.95$ ✓ and ${\text{EVR}}_2=17/20=0.85<0.95$, so $k=2$ is insufficient. $k=3$ is minimal.

---

Recall One-line summary of every cell

A generic cloud ::: ${\lambda }_1\gg {\lambda }_2$, PC1 along the long axis Axis-aligned ::: off-diagonal $=0$, PCs are the original axes Data on a line ::: ${\lambda }_2=0$, rank-1 covariance, lossless to 1D Isotropic tie ::: ${\lambda }_1={\lambda }_2$, PC direction ambiguous Negative covariance ::: PC1 components have opposite signs Different units ::: standardise first or big-unit feature hijacks PC1 Word problem ::: rows = samples, columns = measurements, PC1 = dominant joint trend Choose $k$ ::: smallest $k$ with cumulative EVR $\ge 0.95$

See also: Linear-Algebra-SVD for the numerically-safer route to these same PCs, 2.6.04-Kernel-PCA when a straight line can't capture the cloud, and 3.2.05-Autoencoders for the nonlinear generalisation. For clustering the projected data, 2.5.07-K-Means-Clustering. Foundations in 2.5.01-Introduction-to-Unsupervised-Learning.