Explained variance and choosing components: Exercises

These problems build on the parent topic. Each is labelled L1 → L5 by cognitive demand. Try each one before opening the solution. Every symbol you need was defined in the parent note; a quick reminder of the two workhorses:

$λ_{i}$ = the eigenvalue of principal component $i$ = the amount of variance (spread) the data has along that direction. Bigger $λ_{i}$ = more spread = more information. (See Eigenvalues and Eigenvectors.)

$EVR_{i} = \frac{λ _{i}}{\sum _{j} λ _{j}}$ = the fraction of all the variance living in component $i$ .

$CEV_{k} = \frac{\sum _{i = 1}^{k} λ _{i}}{\sum _{j} λ _{j}}$ = the fraction you keep after taking the first $k$ components.

Level 1 — Recognition

Exercise 1.1

Given eigenvalues $[6, 3, 1]$ , what is the total variance in the data?

Recall Solution

Total variance is just the sum of all eigenvalues (from Step 3 of the parent note: $trace (C) = \sum_{j} λ_{j}$ ). $Total Var = 6 + 3 + 1 = 10$ What we did: added every eigenvalue. Why: the parent derivation showed each $λ_{j}$ is the variance along one principal axis, so their sum is the whole spread.

Exercise 1.2

For the same eigenvalues $[6, 3, 1]$ , compute $EVR_{1}$ as a percentage.

Recall Solution

$EVR_{1} = \frac{6}{10} = 0.60 = 60%$ The first (largest) component alone captures 60% of the spread.

Exercise 1.3

True or false: eigenvalues in a PCA scree plot are always listed in descending order.

Recall Solution

True. PCA sorts components so $λ_{1} \geq λ_{2} \geq \dots \geq λ_{d}$ . That ordering is what makes the Scree Plot slope downward and lets us stop early once returns get small.

Level 2 — Application

Exercise 2.1

A 4D dataset has eigenvalues $[5, 3, 1.5, 0.5]$ . Compute all four $EVR_{i}$ and $CEV_{2}$ .

Recall Solution

Total $= 5 + 3 + 1.5 + 0.5 = 10$ . $EVR_{1} = 0.50, EVR_{2} = 0.30, EVR_{3} = 0.15, EVR_{4} = 0.05$ $CEV_{2} = 0.50 + 0.30 = 0.80 = 80%$ Keeping the first two components retains 80% of the information.

Exercise 2.2

Using the eigenvalues from 2.1, how many components are needed to reach at least 90% cumulative variance?

Recall Solution

March through the cumulative sums: $CEV_{1} = 0.50, CEV_{2} = 0.80, CEV_{3} = 0.95$ $CEV_{3} = 0.95 \geq 0.90$ , so we need $k = 3$ components. Why we stop at 3: the algorithm accumulates variance in importance-order and halts the first time it crosses the threshold — going to $k = 2$ (80%) is not enough, $k = 3$ (95%) clears the bar.

Exercise 2.3

A diagonal Covariance Matrix is $C = 900040002 .$ What fraction of variance is lost if you drop the smallest component?

Recall Solution

For a diagonal covariance matrix the diagonal entries are the eigenvalues: $λ = [9, 4, 2]$ , total $= 15$ . The smallest is $λ_{3} = 2$ , so the fraction lost is $EVR_{3} = \frac{2}{15} \approx 0.1333 = 13.33%.$ You keep $86.67%$ .

Level 3 — Analysis

Exercise 3.1

Two datasets both have total variance $= 20$ .

Dataset A: $λ = [10, 6, 3, 1]$
Dataset B: $λ = [5, 5, 5, 5]$

For a 90% retention target, which dataset compresses better (needs fewer components)? Explain geometrically.

Recall Solution

Dataset A cumulative: $0.50, 0.80, 0.95$ → needs 3 components for 90%. Dataset B cumulative: $0.25, 0.50, 0.75, 1.00$ → needs 4 components for 90% (three give only 75%). A compresses better. Geometrically (see figure below): A's variance is concentrated — the data cloud is a stretched cigar lying mostly along a couple of directions, so a few axes capture almost everything. B's variance is spread evenly — the cloud is a round ball with no preferred direction, so every axis is equally important and you cannot throw any away cheaply.

Figure — Explained variance and choosing components

Exercise 3.2

A dataset has eigenvalues where the top 3 are $[40, 20, 10]$ and the remaining 47 eigenvalues are each exactly $0.4255...$ Actually the remaining 47 sum to $30$ . What is $CEV_{3}$ , and what does a Scree Plot "elbow" tell you here?

Recall Solution

Total $= 40 + 20 + 10 + 30 = 100$ . $CEV_{3} = \frac{40 + 20 + 10}{100} = \frac{70}{100} = 0.70 = 70%.$ The elbow is at component 3: the first three eigenvalues drop steeply ( $40 \to 20 \to 10$ ), then the curve flattens into a long tail of tiny, nearly-equal eigenvalues. That flat tail is the classic signature of noise — many directions each carrying a sliver of variance. The elbow says "structure lives in the first 3; the rest is a haze." (Note the tension: the elbow suggests $k = 3$ but that only gives 70% — thresholds and elbows can disagree.)

Level 4 — Synthesis

Exercise 4.1

You compute PCA two ways on the same centered data matrix $X$ ( $n \times d$ , with $n$ samples): (a) eigen-decomposition of the covariance matrix $C = \frac{1}{n} X^{T} X$ giving eigenvalues $λ_{i}$ ; (b) Singular Value Decomposition (SVD) of $X$ giving singular values $σ_{i}$ . Show the relationship between $λ_{i}$ and $σ_{i}$ , and rewrite $EVR_{i}$ using singular values.

Recall Solution

SVD writes $X = U Σ V^{T}$ . Then $X^{T} X = V Σ^{T} U^{T} U Σ V^{T} = V Σ^{2} V^{T}$ (using $U^{T} U = I$ ). This is exactly the eigen-decomposition of $X^{T} X$ , so its eigenvalues are $σ_{i}^{2}$ , and the covariance eigenvalues are $λ_{i} = \frac{σ _{i}^{2}}{n} .$ Substituting into the ratio, the $\frac{1}{n}$ cancels top and bottom: $EVR_{i} = \frac{λ _{i}}{\sum _{j} λ _{j}} = \frac{σ _{i}^{2} / n}{\sum _{j} σ _{j}^{2} / n} = \frac{σ _{i}^{2}}{\sum _{j} σ _{j}^{2}} .$ Why this matters: you can get explained variance straight from singular values without ever forming the covariance matrix — numerically safer and often faster.

Exercise 4.2

A dataset's singular values are $σ = [10, 8, 6, 2]$ and $n = 4$ samples. Compute the covariance eigenvalues $λ_{i}$ and the total variance, then $CEV_{2}$ .

Recall Solution

Using $λ_{i} = σ_{i}^{2} / n$ with $n = 4$ : $σ^{2} = [100, 64, 36, 4], λ = [25, 16, 9, 1] .$ Total variance $= 25 + 16 + 9 + 1 = 51$ . $CEV_{2} = \frac{25 + 16}{51} = \frac{41}{51} \approx 0.8039 = 80.39%.$ Equivalently via singular values directly: $\frac{100 + 64}{100 + 64 + 36 + 4} = \frac{164}{204} \approx 0.8039$ — identical, as proven in 4.1.

Level 5 — Mastery

Exercise 5.1

Design a retention policy. You have a 50D dataset. Below is a partial cumulative table. Your team needs 95% variance retained, but each kept component costs compute. Given $CEV = [0.42, 0.63, 0.75, 0.83, 0.88, 0.915, 0.94, 0.955, 0.965, \dots]$ (entries are $CEV_{1}, CEV_{2}, \dots$ ), find the smallest $k$ meeting 95%, and argue whether jumping to that $k$ from the previous one is "worth it."

Recall Solution

Scan for the first entry $\geq 0.95$ :

$CEV_{7} = 0.94$ (below), $CEV_{8} = 0.955$ (first to cross).

So the smallest $k$ is $8$ . Marginal analysis: the jump from $k = 7$ to $k = 8$ adds $0.955 - 0.94 = 0.015 = 1.5$ percentage points for one extra component — a modest but real gain that pushes you over the required 95% line. Since the requirement is a hard constraint, that 8th component is mandatory regardless of its small marginal value. Going to $k = 9$ (adds only $0.010$ ) would be optional over-spending, since 95% is already satisfied at $k = 8$ .

Exercise 5.2

Conceptual synthesis. PCA is feature extraction, not feature selection. Suppose two ORIGINAL features $X_{1}, X_{2}$ are perfectly correlated (each has variance 5, correlation $= 1$ ). What does the covariance eigen-structure look like, how much can you compress, and why is this extraction rather than selection?

Recall Solution

With variance 5 each and correlation 1, the covariance $= 5 \cdot 5 = 5$ , so $C = [5555] .$ Eigenvalues: trace $= 10$ , $det = 25 - 25 = 0$ , so $λ = [10, 0]$ .

$EVR_{1} = 10/10 = 100%$ , $EVR_{2} = 0%$ .
You can drop to 1 component with zero information loss — the two features were redundant copies along one line.

Why extraction, not selection: the single surviving component is the direction $\frac{1}{2} (X_{1} + X_{2})$ — a new feature built as a combination of both originals. Dimensionality Reduction by PCA never picks $X_{1}$ or $X_{2}$ ; it manufactures a fresh axis. Feature selection would have kept one raw column and thrown the other away. Here they give the same numerical span, but the mechanism (blend vs. keep-one) is the defining difference.