Principal Component Analysis (PCA) theory

What PCA Actually Does

Principal Component Analysis transforms high-dimensional data into a new coordinate system where:

The first axis (1st principal component) points in the direction of maximum variance
The second axis (2nd PC) points in the direction of maximum remaining variance, orthogonal to the first
Each subsequent axis maximizes variance orthogonal to all previous ones

Why orthogonal? We want components to capture independent sources of variation. Orthogonality ensures no redundancy—each PC adds new information.

The Mathematics: Deriving PCA from Scratch

Given data matrix $\mathbf{X} \in \mathbb{R}^{n \times d}$ ( $n$ samples, $d$ features), first center it: $\mathbf{X}_c = \mathbf{X} - \mathbf{1}_n \boldsymbol{\mu}^T$ where $\boldsymbol{\mu} = \frac{1}{n}\sum_{i=1}^n \mathbf{x}_i$ is the mean vector.

Why center? PCA finds directions through the data's "center of mass". Without centering, the first PC might just point toward the mean instead of capturing variance structure. Centering makes the coordinate origin the data's centroid.

Step 2: Formulate the Variance Maximization Problem

We want to find a unit vector $\mathbf{w}_1$ (first PC) such that projecting data onto it maximizes variance.

Projection of $\mathbf{x}_i$ onto $\mathbf{w}_1$ : $z_i = \mathbf{w}_1^T \mathbf{x}_i$

Variance of projections: $\text{Var}(z) = \frac{1}{n}\sum_{i=1}^n (z_i - \bar{z})^2$

Since data is centered, $\bar{z} = 0$ : $\text{Var}(z) = \frac{1}{n}\sum_{i=1}^n (\mathbf{w}_1^T \mathbf{x}_i)^2 = \frac{1}{n}\sum_{i=1}^n \mathbf{w}_1^T \mathbf{x}_i \mathbf{x}_i^T \mathbf{w}_1$

Why this step? We're expanding the squared scalar projection. Using the property $(\mathbf{a}^T\mathbf{b})^2 = \mathbf{a}^T\mathbf{b}\mathbf{b}^T\mathbf{a}$ when the result is scalar.

Rearranging: $\text{Var}(z) = \mathbf{w}_1^T \left(\frac{1}{n}\sum_{i=1}^n \mathbf{x}_i \mathbf{x}_i^T\right) \mathbf{w}_1 = \mathbf{w}_1^T \mathbf{C} \mathbf{w}_1$

where $\mathbf{C} = \frac{1}{n}\mathbf{X}_c^T\mathbf{X}_c$ is the covariance matrix.

Step 3: Solve with Lagrange Multipliers

We want: $\max_{\mathbf{w}_1} \mathbf{w}_1^T \mathbf{C} \mathbf{w}_1 \quad \text{subject to} \quad \|\mathbf{w}_1\| = 1$

Why the constraint? Without it, we could make variance arbitrarily large by scaling $\mathbf{w}_1$ . The unit constraint means we're finding a direction, not a magnitude.

Lagrangian: $\mathcal{L} = \mathbf{w}_1^T \mathbf{C} \mathbf{w}_1 - \lambda(\mathbf{w}_1^T \mathbf{w}_1 - 1)$

Taking derivative with respect to $\mathbf{w}_1$ and setting to zero: $\frac{\partial \mathcal{L}}{\partial \mathbf{w}_1} = 2\mathbf{C}\mathbf{w}_1 - 2\lambda\mathbf{w}_1 = 0$

Why derivative? We're finding critical points. The factor of 2 appears from differentiating quadratic forms: $\frac{\partial}{\partial \mathbf{w}}(\mathbf{w}^T\mathbf{A}\mathbf{w}) = 2\mathbf{A}\mathbf{w}$ when $\mathbf{A}$ is symmetric.

This gives: $\mathbf{C}\mathbf{w}_1 = \lambda \mathbf{w}_1$

This is the eigen-equation! $\mathbf{w}_1$ must be an eigenvector of $\mathbf{C}$ with eigenvalue $\lambda$ .

Which eigenvector? Substituting back into our objective: $\mathbf{w}_1^T \mathbf{C} \mathbf{w}_1 = \mathbf{w}_1^T (\lambda \mathbf{w}_1) = \lambda \mathbf{w}_1^T \mathbf{w}_1= \lambda$

The variance captured equals $\lambda$ ! To maximize variance, choose the eigenvector with the largest eigenvalue.

Formula PCA Solution

For centered data matrix $\mathbf{X}_c$ , the principal components are:

Compute covariance: $\mathbf{C} = \frac{1}{n}\mathbf{X}_c^T\mathbf{X}_c$
Eigendecomposition: $\mathbf{C} = \mathbf{W}\mathbf{\Lambda}\mathbf{W}^T$
Sort eigenvectors by eigenvalues: $\lambda_1 \geq \lambda_2 \geq \ldots \geq \lambda_d$
Principal components: $\mathbf{w}_1, \mathbf{w}_2, \ldots, \mathbf{w}_d$ (columns of $\mathbf{W}$ )
Projection onto first $k$ PCs: $\mathbf{Z} = \mathbf{X}_c \mathbf{W}_k$ where $\mathbf{W}_k = [\mathbf{w}_1, \ldots, \mathbf{w}_k]$ Variance explained by $k$ components: $\frac{\sum_{i=1}^k \lambda_i}{\sum_{i=1}^d \lambda_i}$

Why Subsequent Components Are Orthogonal

For the second PC $\mathbf{w}_2$ , we maximize variance subject to:

$\|\mathbf{w}_2\| = 1$ (unit length)
$\mathbf{w}_2^T \mathbf{w}_1 = 0$ (orthogonal to first PC)

Mathematical reason: Eigenvectors of a symmetric matrix (like covariance matrices) corresponding to distinct eigenvalues are automatically orthogonal. Since covariance matrices are symmetric real matrices, they have a complete set of orthonormal eigenvectors.

Intuitive reason: If $\mathbf{w}_2$ had any component along $\mathbf{w}_1$ , that component would capture variance already captured by $\mathbf{w}_1$ (redundant). Orthogonality ensures each PC captures new, uncorrelated variance.

Reconstruction and Dimensionality Reduction

Forward Transform (Encoding)

Project data onto first $k$ PCs: $\mathbf{Z} = \mathbf{X}_c \mathbf{W}_k \in \mathbb{R}^{n \times k}$

Each row $\mathbf{z}_i$ is the k-dimensional representation of sample $\mathbf{x}_i$ .

Reconstruction (Decoding)

Approximate original data from reduced representation: $\hat{\mathbf{X}}_c = \mathbf{Z} \mathbf{W}_k^T = \mathbf{X}_c \mathbf{W}_k\mathbf{W}_k^T$

Then add back the mean: $\hat{\mathbf{X}} = \hat{\mathbf{X}}_c + \mathbf{1}_n\boldsymbol{\mu}^T$

Reconstruction error (mean squared error): $\text{MSE} = \frac{1}{n}\|\mathbf{X}_c - \hat{\mathbf{X}}_c\|_F^2 = \sum_{i=k+1}^d \lambda_i$

Why this formula? The Frobenius norm squared equals the sum of eigenvalues we didn't include. Each discarded PC contributes its eigenvalue to the error.

Worked example Worked Example:2D to 1D Reduction

Data: 4 centered points in 2D $\mathbf{X}_c = \begin{bmatrix} 2 & 1 \\ -1 & -0.5 \\ -2 & -1 \\ 1 & 0.5 \end{bmatrix}$

Step 1: Covariance matrix $\mathbf{C} = \frac{1}{4}\mathbf{X}_c^T\mathbf{X}_c = \frac{1}{4}\begin{bmatrix}10 & 5 \\ 5 & 2.5\end{bmatrix} = \begin{bmatrix}2.5 & 1.25 \\ 1.25 & 0.625\end{bmatrix}$

Why this step? Computing $\mathbf{X}_c^T\mathbf{X}_c$ gives us the scatter matrix; dividing by $n$ gives covariance.

Step 2: Eigenvalues $\det(\mathbf{C} - \lambda \mathbf{I}) = (2.5-\lambda)(0.625-\lambda) - 1.25^2 = 0$ $\lambda^2 - 3.125\lambda = 0 \implies \lambda_1 = 3.125, \lambda_2 = 0$

Why this step? The characteristic equation finds values where $\mathbf{C} - \lambda\mathbf{I}$ is singular (has no inverse).

Step 3: Eigenvector for $\lambda_1 = 3.125$ $(\mathbf{C} - 3.125\mathbf{I})\mathbf{w}_1 = 0$ $\begin{bmatrix}-0.625 & 1.25 \\ 1.25 & -2.5\end{bmatrix}\begin{bmatrix}w_{11} \\ w_{12}\end{bmatrix} = 0$

From first row: $-0.625w_{11} + 1.25w_{12} = 0 \implies w_{11} = 2w_{12}$

Normalizing: $\mathbf{w}_1 = \frac{1}{\sqrt{5}}\begin{bmatrix}2 \\ 1\end{bmatrix} \approx \begin{bmatrix}0.894 \\ 0.447\end{bmatrix}$

Why this step? Solving the homogeneous system finds the null space (eigenvector). Normalization ensures unit length.

Step 4: Project $\mathbf{Z} = \mathbf{X}_c \mathbf{w}_1 = \begin{bmatrix}2.236 \ -1.118 \\ -2.236 \\ 1.118\end{bmatrix}$

Interpretation: Each point's position along the principal direction. The first point is2.236 units along the PC.

Variance captured: $\frac{3.125}{3.125 + 0} = 100\%$ (all variance is along one direction—data is perfectly colinear!)

Worked example Example: Choosing Number of Components

You have 50-dimensional data. After PCA, eigenvalues are: $[\lambda_1, \lambda_2, \ldots, \lambda_{50}] = [8.5, 3.2, 1.1, 0.9, 0.6, \ldots, 0.01]$

Cumulative variance explained:

1 PC: $\frac{8.5}{15.3} = 55.6\%$
2 PCs: $\frac{8.5+3.2}{15.3} = 76.5\%$
3 PCs: $\frac{8.5+3.2+1.1}{15.3} = 83.7\%$
4 PCs: $\frac{8.5+3.2+1.1+0.9}{15.3} = 89.5\%$ Decision rule: Keep components until you reach 80-95% variance (domain-dependent). Here, 3-4 components reduce from 50D to 3D/4D while retaining most information.

Why this works? The eigenvalue drop-off shows diminishing returns. Small eigenvalues represent noise or irrelevant variation.

Mathematical Properties

1. PCA Minimizes Reconstruction Error

PCA with $k$ components gives the best rank-k approximation of $\mathbf{X}_c$ under the Frobenius norm: $\mathbf{W}_k = \arg\min_{\mathbf{W}: \mathbf{W}^T\mathbf{W}=\mathbf{I}_k} \|\mathbf{X}_c - \mathbf{X}_c\mathbf{W}\mathbf{W}^T\|_F^2$

Why? This is the Eckart-Young theorem. The best low-rank approximation comes from the SVD/eigendecomposition.

2. PCA via Singular Value Decomposition

Alternative computation: SVD of centered data: $\mathbf{X}_c = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^T$

Then:

Principal components: $\mathbf{W} = \mathbf{V}$ (right singular vectors)
Eigenvalues: $\lambda_i = \frac{\sigma_i^2}{n}$ (squared singular values divided by $n$ )
Projected data: $\mathbf{Z} = \mathbf{U}\mathbf{\Sigma} = \mathbf{X}_c\mathbf{V}$

Why SVD? More numerically stable than computing $\mathbf{X}_c^T\mathbf{X}_c$ explicitly, especially when $d \gg n$ .

3. Covariance in PC Space

After projection, covariance of $\mathbf{Z}$ is diagonal: $\text{Cov}(\mathbf{Z}) = \frac{1}{n}\mathbf{Z}^T\mathbf{Z} = \mathbf{\Lambda}_k = \text{diag}(\lambda_1, \ldots, \lambda_k)$

Meaning: Principal components are uncorrelated. We've decorelated the data.

Common mistake Common Mistake: "PCA Finds the Most Important Features"

Wrong idea: "If I have 100 features and keep the first 3PCs, PCA tells me the 3 most important original features."

Why it feels right: You're reducing from 100 to 3, so it seems like PCA is selecting features.

The truth: PCA creates new features (linear combinations of all originals). The first PC is a weighted sum of all 100 features: $\text{PC}_1 = w_{11}x_1 + w_{12}x_2 + \ldots + w_{1,100}x_{100}$ PCA is feature extraction, not feature selection. Every PC can involve every original feature.

When this matters: If you need interpretability (e.g., "which gene causes this disease?"), PCA components are hard to interpret. Use feature selection methods instead (Lasso, mutual information, etc.).

Common mistake Common Mistake: Forgetting to Center Data

Wrong approach:

pca = PCA(n_components=2)
Z = pca.fit_transform(X)  # X not centered!

Why it feels right: Scikit-learn's PCA centers automatically, so people forget it's necessary.

The problem: Without centering, the first PC often just points from the origin to the data cloud's mean—wasting a component on location rather than variance structure.

Mathematical consequence: The covariance matrix would be $\mathbb{E}[\mathbf{x}\mathbf{x}^T]$ instead of $\mathbb{E}[(\mathbf{x}-\boldsymbol{\mu})(\mathbf{x}-\boldsymbol{\mu})^T]$ . The former includes mean offset.

The fix: Always center before PCA (or verify your library does it):

X_centered = X - X.mean(axis=0)

Common mistake Common Mistake: Applying PCA After Scaling Different Features Differently

Scenario: You have features in different units (age in years, income in dollars, height in cm). You apply PCA directly.

The problem: PCA is not scale-invariant. Features with larger numeric ranges dominate the variance:

Income: variance ≈ 10^9 (dollars squared)
Age: variance ≈ 100 (years squared) The first PC will mostly capture income variation, ignoring age.

Why this matters: The "most variance" direction depends on arbitrary unit choices. Age in days vs. years changes the result!

The fix: Standardize (z-score) features before PCA when they have different scales: $x'_{ij} = \frac{x_{ij} - \mu_j}{\sigma_j}$

Alternative view: Standardizing means performing PCA on the correlation matrix instead of covariance matrix.

Recall Explain to a 12-Year-Old

Imagine your class takes a photo together. You have everyone's position in3D space (left-right, forward-back, up-down). But when you look at the photo, you realize most of the information is in just 2 dimensions—the photo plane! The 3rd dimension (depth) barely matters because everyone was standing at roughly the same distance from the camera.

PCA is like finding the "photo angle" for data. It looks at your data cloud and asks: "If I had to take a picture (project onto fewer dimensions), which angle keeps the most information?" The first principal component is like the camera's main direction—it captures where people are most spread out. The second PC is like looking from the side to catch any remaining spread.

Why do we want fewer dimensions? Imagine describing everyone's position. Instead of saying "3.2m left, 1.8m forward, 0.1m higher than Sarah," you could just say "Row 3, Position 7" if everyone's standing in neat rows. Two numbers instead of three! PCA finds those "neat rows" in mesy data.

Connections

Covariance Matrix — PCA decomposes this to find variance structure
Eigendecomposition — The mathematical tool underlying PCA
Singular Value Decomposition — Alternative (more stable) way to compute PCA
Linear Autoencoders — Neural networks that learn PCA-like representations when linear
Kernel PCA — Nonlinear extension using the kernel trick
Factor Analysis — Probabilistic model related to PCA with explicit noise
Independent Component Analysis — Finds statistically independent (not just uncorrelated) components
Feature Scaling — Necessary preprocessing for PCA with mixed units
Dimensionality Reduction — PCA is one technique; compare with t-SNE, UMAP
Variance — The quantity PCA maximizes in successive orthogonal directions

#flashcards/ai-ml

What is the goal of PCA? :: Find orthogonal directions that maximize variance in the data, enabling dimensionality reduction while preserving the most information.

Why must we center data before PCA?

So the first PC captures variance structure rather than just pointing toward the mean; PCA seeks variance around the centroid, not from the origin.

What is a principal component mathematically?

An eigenvector of the covariance matrix; the k-th PC is the eigenvector corresponding to the k-th largest eigenvalue.

What does the eigenvalue represent in PCA?

The variance captured by the corresponding principal component; larger eigenvalue = more variance explained.

How do we compute the covariance matrix for PCA?

\mathbf{C} = \frac{1}{n}\mathbf{X}_c^T\mathbf{X}_c

where

\mathbf{X}_c

is the centered data matrix (rows are samples, columns are features).

Why are principal components orthogonal?

(1) Eigenvectors of symmetric matrices with distinct eigenvalues are orthogonal. (2) Ensures each PC captures independent variance, no redundancy.

How do we project data onto the first k principal components?

\mathbf{Z} = \mathbf{X}_c \mathbf{W}_k

where

\mathbf{W}_k

contains the first k eigenvectors as columns.

What is the reconstruction formula in PCA?

\hat{\mathbf{X}}_c = \mathbf{Z}\mathbf{W}_k^T = \mathbf{X}_c\mathbf{W}_k\mathbf{W}_k^T

, then add back mean:

\hat{\mathbf{X}} = \hat{\mathbf{X}}_c + \mathbf{1}_n\boldsymbol{\mu}^T

What is the reconstruction error when keeping k components?

\text{MSE} = \sum_{i=k+1}^d \lambda_i

— the sum of discarded eigenvalues equals the squared error.

How much variance do k components explain?

\frac{\sum_{i=1}^k \lambda_i}{\sum_{i=1}^d \lambda_i}

— ratio of sum of kept eigenvalues to total sum.

Is PCA feature selection or feature extraction?

Feature extraction; it creates new features (linear combinations of all originals), not selecting a subset.

Why standardize features before PCA?

Because PCA is not scale-invariant; features with larger numeric scales dominate variance. Standardizing puts all features on equal footing.

What is the difference between covariance-based and correlation-based PCA?

Covariance PCA uses raw data; correlation PCA uses standardized data (equivalent to computing eigenvectors of correlation matrix). Use correlation when features have different units.

How does SVD relate to PCA?

\mathbf{X}_c = \mathbf{U\Sigma V}^T

; principal components are right singular vectors

\mathbf{V}

, eigenvalues are

\lambda_i = \sigma_i^2/n

. More numerically stable.

What is the covariance structure in PC space?

Diagonal —

\text{Cov}(\mathbf{Z}) = \mathbf{\Lambda}_k

. Principal components are uncorrelated by construction.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

PCA ka matlab hai Principal Component Analysis – yek technique hai jisse hum high-dimensional data ko kam dimensions mein convert kar sakte hain, lekin important information retain rahti hai. Socho ki tumhare pas 100 features hain (100 columns), lekin asli mein sirf 3-4 "directions" mein sab variation hai. PCA un important directions ko dhondh leta hai.

Mathematically, PCA covariance matrix ke eigenvectors nikalta hai. Covariance

Principal Component Analysis (PCA) theory

What PCA Actually Does

The Mathematics: Deriving PCA from Scratch

Step 1: Center the Data (WHY)

Step 2: Formulate the Variance Maximization Problem

Step 3: Solve with Lagrange Multipliers

Why Subsequent Components Are Orthogonal

Reconstruction and Dimensionality Reduction

Forward Transform (Encoding)

Reconstruction (Decoding)

Mathematical Properties

1. PCA Minimizes Reconstruction Error

2. PCA via Singular Value Decomposition

3. Covariance in PC Space

Connections

Concept Map

Hinglish (regional understanding)

Test yourself — Unsupervised Learning

Connections