2.5.10 · D5Unsupervised Learning
Question bank — Explained variance and choosing components
This is a D5 page: no heavy arithmetic, just the ideas that break people's mental models of explained variance and choosing components. Each item is a one-line reveal. Cover the answer, commit to a guess, then check — the justification is the point, not the yes/no.
Prerequisites worth re-reading if a question stumps you: Eigenvalues and Eigenvectors, Covariance Matrix, Scree Plot, Singular Value Decomposition (SVD), Dimensionality Reduction, Feature Selection vs. Feature Extraction.
Notation you need before answering
Everything below leans on three objects. Pin them down first so no symbol is a mystery.
The two pictures below are the ones the "scree plot" and "reconstruction" traps refer to — glance at them before those items so the words point at something concrete.
True or false — justify
The eigenvalue of the covariance matrix equals the variance of the data projected onto principal component .
True — projecting onto a unit eigenvector gives variance , so the eigenvalue is that component's variance, not merely proportional to it.
Explained variance ratios of all components always sum to exactly .
True — the denominator is the sum of all eigenvalues, so summing every gives ; nothing is lost until you drop a component.
If the first component explains of variance, keeping it retains of the useful information for a downstream classifier.
False — variance is spread, not task-relevant information; the discarded can hold the very direction that separates your classes.
Cumulative explained variance can decrease as increases.
False — each added component contributes a non-negative eigenvalue, so is monotonically non-decreasing and flattens toward .
Total variance equals the sum of the covariance matrix's eigenvalues.
True — total variance is , and the trace of any matrix equals the sum of its eigenvalues, so the two are identical.
If two eigenvalues are equal, the choice of which pair of eigenvectors to call "component 1" and "component 2" is unique.
False — equal eigenvalues span a degenerate subspace, so any orthonormal basis of that plane is equally valid; the individual directions are arbitrary though the retained subspace is fixed.
A component with a tiny eigenvalue is always pure noise and safe to drop.
False — usually it is low-information, but a genuinely rare, low-variance signal can carry crucial structure; "small variance" is a heuristic for noise, not a proof.
Explained variance ratio changes if you scale one feature to different units (e.g. metres → millimetres).
True — rescaling a feature inflates its variance, dominating the covariance matrix and skewing every ; this is why standardising features before PCA is standard.
Keeping components until guarantees the reconstructed data looks correct pixel-by-pixel.
False — it guarantees of variance (squared spread) is retained; reconstruction error is measured in the same squared-norm sense, and perceptual quality is a separate matter.
Singular values from SVD of centered relate to eigenvalues by .
True — with the covariance convention, ; using instead just swaps the divisor, so squaring singular values recovers the eigenvalues up to that fixed scale factor.
Spot the error
", the second eigenvalue over the first."
Wrong denominator — it must be over the total , not ; measures a share of all variance, not a ratio to the top component.
"I standardised nothing and one feature is in millions, but PCA is scale-free so it's fine."
PCA is not scale-free — the giant-variance feature will hijack component 1; you must standardise (or use correlation instead of covariance) first.
"The scree plot's elbow is at component 7, so I keep components 8 through the end."
Backwards — the elbow marks where returns diminish, so you keep components up to and including the elbow (roughly 1–7) and drop the flat tail after it.
"Variance explained is negative for the last component because it removes information."
Impossible — eigenvalues of a covariance matrix are , so every ; components only add non-negative variance.
"To reach variance I picked the 90 largest-index components."
Confused index with magnitude — you sort eigenvalues descending and take the first few largest ones, not the ones with big position numbers.
"CEV hit at , but let me also add component 10 since its eigenvalue is nonzero."
Defeats the purpose — once the threshold is met you stop; adding a low- component barely raises CEV while costing you the compression you wanted.
"This is feature selection: PCA picks the 3 best original columns."
No — PCA is [[Feature Selection vs. Feature Extraction|feature extraction]]; components are new linear combinations of all original features, not a subset of them.
Why questions
Why do we divide by the sum of all eigenvalues rather than the number of dimensions ?
Because we want a fraction of total variance, and total variance is (the trace), not ; dividing by would give an average, answering a different question.
Why is the eigenvalue interpretable as variance in the first place?
The covariance matrix measures spread, and projecting data onto an eigenvector diagonalises so that direction's spread is exactly its eigenvalue — variance and eigenvalue coincide there.
Why prefer cumulative explained variance over looking at one component at a time?
Because information is distributed across components; a single can look tiny yet the accumulation of several such directions preserves the data's structure — the sum is what you actually retain.
Why does an image typically need hundreds of components while toy 3D data needs one or two?
Real images have variance thinly spread over many directions (rich detail), so no single component dominates; simple data concentrates variance in a few directions, letting a couple of components capture almost everything.
Why can two datasets with identical CEV curves demand different component counts for a task?
Because CEV measures variance retention, blind to which variance is task-relevant; a low-variance-but-discriminative direction can force you to keep more components than variance alone suggests.
Why does standardising features before PCA change the answer?
PCA maximises variance, and variance depends on units; standardising puts every feature on equal footing so the components reflect correlation structure rather than whichever feature happened to have the largest raw scale.
Edge cases
What is if the covariance matrix is the zero matrix (all points identical)?
Undefined — total variance is , so the denominator vanishes; there is no spread to explain and PCA has nothing to reduce.
If all eigenvalues are equal (perfectly isotropic data), how many components should you keep for variance?
About — every direction carries equal variance , so there is no compression to gain; PCA offers no advantage on isotropic data.
What does it mean when several trailing eigenvalues are exactly zero?
Those directions have no spread, so the data actually lives in a lower-dimensional subspace; the rank of is the true dimensionality and you can drop the zero- components losslessly.
If (fewer samples than features), how many nonzero eigenvalues can the covariance matrix have at most?
At most (after centering) — the data cannot span more directions than it has independent centered points, so beyond that eigenvalues are forced to zero.
For a single component whose , what does the data geometry look like?
All variance lies along one line — the points are collinear (up to the mean), so one component reconstructs everything exactly and every other eigenvalue is zero.
What happens to if you accidentally include a constant (zero-variance) feature?
It contributes to both numerator and denominator, so it never affects — but it wastes a dimension and can signal a data-cleaning bug worth fixing.
Recall Quick self-check
Which single number tells you the total variance to divide by? ::: The trace of the covariance matrix, equal to . Is CEV ever non-monotonic? ::: No — it only increases (or stays flat) as you add non-negative eigenvalues. Does high CEV guarantee good downstream task accuracy? ::: No — variance retention and task-relevant information are different things.