PCA via eigendecomposition and SVD: Question bank

Before we start, a tiny shared vocabulary so no symbol appears unearned:

Definition Words we reuse everywhere below

==Number of samples $n$ ::: how many data points (rows) you have; number of features $d$ == ::: how many measurements (columns) each point carries.
Centering ::: subtracting the mean of each feature so the data cloud sits around the origin.
Variance along a direction ::: how spread out the points are when you shine a flashlight and read their shadows on a line pointing that way (see the figure below). Big spread = big variance.
Principal component (PC) ::: a unit-length direction $v$ (an arrow of length 1) that PCA chooses because the shadows on it are maximally spread.
==Eigenvalue $λ_{i}$ == ::: the amount of variance living along PC number $i$ . Bigger $λ$ = more important direction.
==Covariance matrix $C$ == ::: the $d \times d$ table whose diagonal holds each feature's variance and whose off-diagonals hold how pairs of features move together.
==Trace of $C$ == ::: simply the sum of the diagonal entries of $C$ — i.e. the total variance summed across all features.
==Singular value $σ_{i}$ == ::: what SVD produces instead of eigenvalues; related by $λ_{i} = σ_{i}^{2} / (n - 1)$ , where $n$ is the number of samples.
== $V_{k}$ == ::: the $d \times k$ matrix whose columns are the top $k$ eigenvectors (PCs), i.e. $V_{k} = [v_{1} ∣ \dots ∣ v_{k}]$ ; projecting onto it does the dimensionality reduction.

Figure — PCA via eigendecomposition and SVD

Look at the figure: the grey dots are the data cloud, the two arrows are the PCs $v_{1}$ (long, big $λ_{1}$ ) and $v_{2}$ (short, small $λ_{2}$ ), and the coloured tick marks on the long arrow are the shadows (projections) — their spread is the variance along $v_{1}$ .

True or false — justify

PCA components are always orthogonal (at right angles) to each other.

True.

C

is symmetric, and symmetric matrices have eigenvectors that can always be chosen mutually orthogonal — so the PCs form a clean set of perpendicular axes.

The first principal component always passes through the origin.

True after centering — because we subtract the mean, the cloud is centered at the origin and the PC is a direction (arrow from origin), not a line floating in space.

If two features are already uncorrelated but have very different variances, PCA cannot compress them.

False — it still compresses. With unequal variances

C

is diagonal but not a scaled identity; the feature with the larger variance becomes PC 1 and the smaller one can be dropped. Compression only fails in the special case where the variances are also equal (a scaled identity), because then every direction is equally "principal."

The eigenvalues of the covariance matrix can be negative.

False.

C

is positive semi-definite (it is a sum of squares), so every eigenvalue

λ_{i} \geq 0

; a "negative variance" is meaningless.

Doubling the units of one feature (e.g. cm → mm) leaves the principal components unchanged.

False. Multiplying a feature by 10 multiplies its variance by 100, so it hijacks the first PC. This is exactly why we often scale features first.

The projection

Z = X_{centered} V_{k}

(with

V_{k}

the top-

k

eigenvector matrix) reduces the number of samples.

False. It reduces the number of columns (features

d \to k

); the number of rows (samples

n

) is untouched — each sample gets a shorter description, none disappear.

SVD and eigendecomposition of

C

give the same principal components.

True. Substituting

X_{centered} = U Σ V^{T}

into

C

shows the right singular vectors

V

are the eigenvectors of

C

, with

λ_{i} = σ_{i}^{2} / (n - 1)

A principal component and its negative (

v

- v

) represent different directions of variance.

False. They describe the same line of spread; sign is arbitrary because

C (- v) = λ (- v)

too. Different software may flip signs — this is not a bug.

PCA can only ever find directions that are straight lines through the data.

True for standard PCA. It is a linear method. For curved (nonlinear) structure you need Kernel PCA or autoencoders.

Spot the error

"I skipped centering because my data was already roughly in the positive quadrant."

Error: roughly positive ≠ centered at origin. Without subtracting the mean, the top PC points from the origin toward the mean — encoding location, not spread. Always center.

"I computed

C = X^{T} X / (n - 1)

using the raw uncentered

X

Error: covariance requires the centered matrix. Using raw

X

gives the second-moment matrix, whose top eigenvector is contaminated by the mean direction.

"I picked the eigenvector with the smallest eigenvalue as my first principal component to keep the important info."

Error: smallest

λ

= least variance = least information (often just noise). The first PC is the largest eigenvalue's eigenvector.

"My dataset has

n = 5

samples and my top singular value came out as

σ_{1} = 5.72

, so the variance explained is

5.72

Error: the variance is

λ_{1} = σ_{1}^{2} / (n - 1) = 5.7 2^{2} / (5 - 1) \approx 32.72/4 = 8.18

, not

σ_{1}

itself. You must square and divide by

n - 1

"To decorrelate features I kept

C

's off-diagonal terms."

Error: decorrelation means removing off-diagonal covariance. Projecting onto the eigenvectors makes the new covariance matrix diagonal — that is the point.

"I formed the huge

d \times d

covariance matrix for my dataset with

d = 10000

features and

n = 50

samples."

Error: wasteful and numerically shaky. With

n ≪ d

, use SVD directly on

X_{centered}

— it never builds the

10000 \times 10000

matrix and is more stable.

"Explained variance ratio for the top 2 PCs was 91%, so I have 91% of the samples."

Error: EVR is about variance (spread), not sample count. 91% means the 2D shadow preserves 91% of the cloud's total spread; all samples remain.

Why questions

Why do we divide by

n - 1

and not

n

(with

n

the number of samples) when forming

C

Because we already spent one "degree of freedom" estimating the mean to center the data; dividing by

n - 1

(Bessel's correction) makes the variance estimate unbiased.

Why does maximizing

v^{T} Cv

over unit-length directions

v

(with

∥ v ∥ = 1

) force

v

to be an eigenvector?

The Lagrange condition gives

Cv = λ v

exactly — the definition of an eigenvector. So the best-spread direction has to be one of

C

's eigenvectors.

Why is the eigenvalue equal to the variance along its PC

v

, not just proportional?

Plugging the eigenvector back in:

Var = v^{T} Cv = v^{T} (λ v) = λ ∥ v ∥^{2} = λ

, since

∥ v ∥ = 1

. The unit-length constraint makes it exact.

Why can SVD avoid the numerical trouble that eigendecomposition of

C

suffers?

Forming

X^{T} X

squares the condition number, amplifying rounding error. SVD works on

X_{centered}

directly, so the small singular values stay resolvable.

Why order the components by eigenvalue at all?

So we can greedily keep the top few and know exactly how much spread we're keeping — the ordering turns "which axes matter" into a simple ranked list.

Why is PCA called an unsupervised method?

It uses no labels or targets — it only looks at how the input features spread and co-vary, which is why it lives under Unsupervised Learning alongside K-Means.

Why does

trace (C) = \sum_{i} λ_{i}

(the sum of

C

's diagonal entries equalling the sum of eigenvalues) let us compute EVR without ever finding eigenvectors?

The total variance (sum of diagonal variances) equals the sum of eigenvalues, so denominators in EVR are cheap — you can get "how much total spread" from the trace alone.

Edge cases

What happens when all data points are identical?

The centered matrix is all zeros,

C = 0

, every eigenvalue is 0. There is no variance to explain — PCA returns nothing meaningful.

What if two eigenvalues are exactly equal (a "tie")?

The two PCs are not uniquely defined — any orthonormal pair spanning that plane works equally well. The spread is circular in that plane, so no single "best" axis exists.

What if you request more components

k

than the data's true rank?

The extra components have eigenvalue 0 and point in arbitrary directions within the null space; they add columns of essentially noise to

Z

and explain no variance.

What if

n < d

(fewer samples than features)?

The covariance matrix is singular with at most

n - 1

nonzero eigenvalues; you can find at most

n - 1

meaningful PCs, and SVD is the sensible route.

What happens to PCA if one feature is a perfect copy of another?

The two columns are linearly dependent, making

C

rank-deficient — one eigenvalue is exactly 0, and PCA correctly collapses the redundant pair into a single direction.

What does the last principal component (smallest

λ

) tell you?

The direction of least spread — often a near-constant relationship or pure noise. It's what you throw away when compressing, but it can reveal hidden constraints in the data.

If every feature already has variance 1 and zero correlation, what does PCA return?

C = I

, so all eigenvalues equal 1 and every direction is equally principal. PCA gives no preferred axes and no compression is possible without loss.

Recall Quick self-check

One-line trap that catches most people ::: Confusing the singular value $σ_{i}$ with the variance — remember variance is $σ_{i}^{2} / (n - 1)$ , never $σ_{i}$ itself.