Explained variance and choosing components

Core Question

When we run PCA and get principal components, how many should we keep? Too few and we lose information; too many and we defeat the purpose of dimensionality reduction. Explained variance gives us a quantitative way to answer this.

[!intuition] The Big Picture

Imagine you're compressing a 100-dimensional dataset. Each principal component captures some amount of the original "spread" (variance) in the data. The first component captures the most variance, the second captures the next most (from what's left), and so on.

The key insight: We don't need components that explain almost no variance—they're mostly noise. We want to keep just enough components to capture, say, 95% of the total variance. This balances compression with information retention.

Why variance matters: In data space, variance = information. Directions with high variance are where data points differ meaningfully. Low-variance directions are often just noise or irrelevant detail.

[!definition] Explained Variance

The explained variance ratio for the $i$-th principal component is:

$\text{EVR}_i = \frac{\lambda_i}{\sum_{j=1}^{d} \lambda_j}$

where:

• $\lambda_i$ is the eigenvalue (variance) of the $i$-th principal component
• $d$ is the original number of dimensions
• The denominator is the total variance in the original data

What it means: $\text{EVR}_i$ tells us what fraction of the total variance is captured by component $i$.

The cumulative explained variance for the first $k$ components is:

$\text{CEV}_k = \sum_{i=1}^{k} \text{EVR}_i = \frac{\sum_{i=1}^{k} \lambda_i}{\sum_{j=1}^{d} \lambda_j}$

This tells us what fraction of total variance we've captured if we keep the first $k$ components.

[!formula] Derivation from First Principles

Step 1: What is variance in the original data?

The total variance in the original data matrix $\mathbf{X}$ (centered, $n \times d$) is the sum of variances along each original dimension:

$\text{Total Var} = \sum_{j=1}^{d} \text{Var}(X_j) = \sum_{j=1}^{d} \frac{1}{n}\sum_{i=1}^{n} (x_{ij} - \bar{x}_j)^2$

If $\mathbf{X}$ is already centered ($\bar{x}_j = 0$), this simplifies to:

$\text{Total Var} = \frac{1}{n} \|\mathbf{X}\|_F^2$

where $\|\cdot\|_F$ is the Frobenius norm (sum of all squared entries).

Why this step? We need a baseline: what's the total "spread" in the original data that we're trying to preserve?

Step 2: Variance captured by a principal component

When we project data onto principal component $\mathbf{v}_i$ (an eigenvector of the covariance matrix), the variance of the projected data is:

$\text{Var}(\mathbf{X}\mathbf{v}_i) = \frac{1}{n} (\mathbf{X}\mathbf{v}_i)^T (\mathbf{X}\mathbf{v}_i) = \frac{1}{n} \mathbf{v}_i^T \mathbf{X}^T \mathbf{X} \mathbf{v}_i$

Since the covariance matrix is $\mathbf{C} = \frac{1}{n}\mathbf{X}^T\mathbf{X}$ and $\mathbf{v}_i$ is an eigenvector with eigenvalue $\lambda_i$:

$\mathbf{C}\mathbf{v}_i = \lambda_i \mathbf{v}_i$

We get:

$\text{Var}(\mathbf{X}\mathbf{v}_i) = \frac{1}{n} \mathbf{v}_i^T (n \mathbf{C}) \mathbf{v}_i = \mathbf{v}_i^T \mathbf{C} \mathbf{v}_i = \mathbf{v}_i^T (\lambda_i \mathbf{v}_i) = \lambda_i \mathbf{v}_i^T \mathbf{v}_i = \lambda_i$

(Since eigenvectors are normalized: $\mathbf{v}_i^T \mathbf{v}_i = 1$)

Why this step? The eigenvalue $\lambda_i$ is the variance along component $i$. This is the fundamental connection between eigenvalues and explained variance.

Step 3: Total variance equals sum of eigenvalues

A beautiful property of covariance matrices:

$\text{Total Var} = \text{trace}(\mathbf{C}) = \sum_{j=1}^{d} \lambda_j$

Why? The trace of a matrix equals the sum of its eigenvalues (a linear algebra identity). The trace of the covariance matrix is also the sum of variances along the original axes.

Why this step? This gives us the denominator in our explained variance formula. The total variance in the data equals the sum of all eigenvalues.

Step 4: Putting it together

The fraction of variance explained by component $i$:

$\text{EVR}_i = \frac{\text{Variance of component } i}{\text{Total variance}} = \frac{\lambda_i}{\sum_{j=1}^{d} \lambda_j}$

For the first $k$ components combined:

$\text{CEV}_k = \frac{\lambda_1 + \lambda_2 + \cdots + \lambda_k}{\sum_{j=1}^{d} \lambda_j}$

Why this step? This ratio tells us the information retention: if $\text{CEV}_k = 0.95$, we've kept 95% of the variance (information) while reducing from $d$ to $k$ dimensions.

[!example] Worked Example 1: 3D to 2D Reduction

Setup: We have 3D data with covariance matrix:

$\mathbf{C} = \begin{bmatrix} 4 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & .1 \end{bmatrix}$

(Diagonal for simplicity; in general, we'd compute eigenvectors of a non-diagonal $\mathbf{C}$)

Eigenvalues: $\lambda_1 = 4, \lambda_2 = 1, \lambda_3 = 0.1$

Step 1: Total variance $\sum_{j=1}^{3} \lambda_j = 4 + 1 + 0.1 = 5.1$

Why this step? This is our baseline—100% of the variance in the original 3D data.

Step 2: Explained variance ratios $\text{EVR}_1 = \frac{4}{5.1} \approx 0.784 \quad (78.4\%)$ $\text{EVR}_2 = \frac{1}{5.1} \approx 0.196 \quad (19.6\%)$ $\text{EVR}_3 = \frac{0.1}{5.1} \approx 0.020 \quad (2.0\%)$

Why this step? Each ratio shows how much information one component carries.

Step 3: Cumulative explained variance $\text{CEV}_1 = 0.784$ $\text{CEV}_2 = 0.784 + 0.196 = 0.980 \quad (98\%)$ $\text{CEV}_3 = 0.980 + 0.020 = 1.000 \quad (100\%)$

Why this step? This tells us: if we keep 2 components, we retain 98% of the variance.

Decision: Keep 2 components. We lose only 2% of the information but reduce dimensionality by 33%.

[!example] Worked Example 2: Image Compression

Setup: A 64×64 grayscale image flattened to 4096 dimensions. After PCA, the first 10 eigenvalues are:

$[1200, 800, 400, 200, 150, 100, 50, 30, 20, 10]$

Total eigenvalues sum to 15,000.

Step 1: EVR for first component $\text{EVR}_1 = \frac{1200}{15000} = 0.08 \quad (8\%)$

Why this step? Even the best component only captures 8% of variance—images are complex!

Step 2: Cumulative for first 5 components $\text{CEV}_5 = \frac{1200 + 800 + 400 + 200 + 150}{15000} = \frac{2750}{15000} \approx 0.183 \quad (18.3\%)$

Why this step? 5 components capture less than 20%—not enough for reconstruction.

Step 3: How many for 90% variance?

We need $\text{CEV}_k \geq 0.90$, so $\sum_{i=1}^{k} \lambda_i \geq 0.90 \times 15000 = 13500$.

Suming eigenvalues (in practice, we'd have all of them and compute cumulative sums), suppose we find need the first 200 components to reach 13,500.

Why this step? This is the practical cutoff—we can compress from 4096 to 200 dimensions (95% reduction) while keeping 90% of image detail.

[!example] Worked Example 3: Choosing $k$ with a Threshold

Setup: We run PCA on a 50-dimensional dataset. We get eigenvalues in descending order. We want to keep enough components to retain 95% of variance.

Algorithm:

cumsum = 0
total_var = sum(eigenvalues)
threshold = 0.95 * total_var
k = 0
 
for i, eigenvalue in enumerate(eigenvalues):
    cumsum += eigenvalue
    k = i + 1
    if cumsum >= threshold:
        break

Why this step? We iterate through components in order of importance, accumulating variance until we hit our target.

Example numbers: If eigenvalues are [10, 8, 5, 3, 2, 1, 0.5, ...] (total = 50):

• After 1 component: 10/50 = 20%
• After 2: 18/50 = 36%
• After 3: 23/50 = 46%
• After 4: 26/50 = 52%
• Continue until cumulative ≥ 47.5 (which is 95% of 50)

Why this approach? It's automatic and data-driven. No guessing, just a clear retention target.

[!mistake] Common Pitfalls

Mistake 1: "The first component explains 60%, so I should keep just that one"

Why it feels right: 60% sounds like a majority—more than half!

What's wrong:

• In high-dimensional data, you often need multiple components for good reconstruction.
• 60% variance ≠ 60% of the "useful information." The remaining 40% might contain crucial structure for downstream tasks (e.g., classification).
• Standard practice: aim for 90-95% cumulative variance, not just the first component.

The fix: Always look at cumulative explained variance and set a threshold (e.g., 95%). Use a scree plot (eigenvalues vs. component number) to see where returns diminish.

Steel-man: The mistake comes from treating variance like a simple percentage score. In reality, information is distributed across components, and we need to aggregate enough of them to preserve the data's structure.

Mistake 2: "Eigenvalues are small after component 10, so those components are useless"

Why it feels right: Small eigenvalues mean low variance—seems like noise.

What's wrong:

• "Small" is relative. If $\lambda_{11} = 0.01$ but the total variance is 0.5, that's 2% of the total—not nothing!
• Sometimes lower components capture rare but important features (e.g., outlier detection, fine-grained classes).

The fix: Use relative measures (explained variance ratio, cumulative %), not absolute eigenvalue magnitudes. Also consider your task: for visualization, 2-3 components suffice; for reconstruction, you might need many more.

Mistake 3: "More components = better, so I'll keep 90% of the original dimensions"

Why it feels right: More information is better, and we avoid data loss.

What's wrong:

• You defeat the purpose of PCA! If you keep 90% of dimensions, you've barely reduced complexity.
• The last components often capture noise, not signal. Including them hurts generalization (overfitting).

The fix: PCA is about compression and noise reduction. Agressively drop low-variance components. A good rule: keep $k$ such that $\text{CEV}_k \geq 0.95$ and $k \ll d$ (e.g., $k \leq 0.1d$ for real compression).

[!recall]- Explain to a 12-Year-Old

Imagine you have a big pile of 100 different measurements about students: height, weight, test scores, hours studied, favorite color coded as a number, shoe size, and94 more weird things.

Now, some of these measurements are really important for understanding students (like test scores), and some are pretty useless (like shoe size for predicting grades).

PCA is like a smart organizer that creates new "super-measurements" (principal components). The first super-measurement combines the old measurements in a way that captures the MOST differences between students. The second captures the next most, and so on.

Here's the key: explained variance tells you how much of the "differences between students" each super-measurement captures.

• Component 1 might capture 40% of the differences
• Component 2 might capture 25%
• Component 3 might capture 15%
• ...and so on

Now, instead of tracking all 100 measurements, you can just keep the first few super-measurements that add up to, say, 95% of the differences. Maybe that's only 10components! You've gone from 100 numbers per student to 10, while keeping95% of the information.

How do you choose? You add up the percentages (cumulative explained variance) until you hit your goal (like 95%). That's how many components you keep.

[!mnemonic] The Scree Plot Elbow

"Find the elbow where the scree falls off the cliff"

A scree plot shows eigenvalues (y-axis) vs. component number (x-axis). It looks like a mountain with a steep drop and then a long flat tail.

Mnemonic: Imagine pouring scree (loose rocks) down a mountainside. The big boulders (high eigenvalues) tumble down first and pile up. Then smaller pebbles (low eigenvalues) dribble down and spread out flat at the bottom.

You want to keep components before the plot flattens out—that's where the "information boulders" are. The flat tail is just "noise pebbles."

Look for the elbow (the bend where steep becomes flat). Keep components up to the elbow.

Choosing $k$ in Practice

Method 1: Cumulative Variance Threshold

• Choose a target (e.g., 95% or 99%)
• Keep smallest $k$ such that $\text{CEV}_k \geq$ target
• Best for: Reconstruction, compression tasks

Method 2: Scree Plot Elbow

• Plot eigenvalues vs. component index
• Find the "elbow" where the curve flattens
• Keep components before the elbow
• Best for: Exploratory analysis, visualization

Method 3: Cross-Validation (Supervised Tasks)

• For each $k$, reduce to $k$ dimensions
• Train a model on reduced data
• Validate performance
• Choose $k$ that maximizes validation accuracy
• Best for: Classification, regression preprocessing

Method 4: Fixed Dimension Target

• Decide $k$ based on constraints (e.g., "I need 2D for a plot" or "I can only store 50 dimensions")
• Check $\text{CEV}_k$ to see how much variance you're retaining
• Best for: Visualization (2D/3D), storage/speed constraints

Key Formulas Summary

| Concept | Formula | Interpretation | |---------|-------------| | Explained Variance Ratio | $\text{EVR}_i = \frac{\lambda_i}{\sum_{j=1}^{d} \lambda_j}$ | Fraction of variance captured by component $i$ | | Cumulative Explained Variance | $\text{CEV}_k = \sum_{i=1}^{k} \text{EVR}_i$ | Total fraction captured by first $k$ components | | Total Variance | $\sum_{j=1}^{d} \lambda_j = \text{trace}(\mathbf{C})$ | Sum of all eigenvalues = trace of covariance | | Component Variance | $\text{Var}(\mathbf{X}\mathbf{v}_i) = \lambda_i$ | Eigenvalue = variance along that component |

Connections

• Principal Component Analysis (PCA): Explained variance is how we decide how many components to keep from PCA
• Eigenvalues and Eigenvectors: Eigenvalues directly represent the variance captured by each principal component
• Covariance Matrix: The trace of the covariance matrix equals the total variance in the data
• Dimensionality Reduction: Explained variance quantifies the information loss when we reduce dimensions
• Scree Plot: Visual tool for choosing $k$ by finding the "elbow"
• Singular Value Decomposition (SVD): Singular values are related to eigenvalues; $\sigma_i^2 = n \lambda_i$ for PCA via SVD
• Bias-Variance Tradeoff: Keeping too many components increases variance (overfitting); too few increases bias
• Feature Selection vs. Feature Extraction: Explained variance helps decide how many extracted features (components) to use

#flashcards/ai-ml

What is the explained variance ratio for principal component $i$? :: $\text{EVR}_i = \frac{\lambda_i}{\sum_{j=1}^{d} \lambda_j}$, the fraction of total variance captured by component $i$, where $\lambda_i$ is its eigenvalue.

Concept Map

Hinglish (regional understanding)