Singular Value Decomposition (SVD) — full derivation: Question bank

Every item leans on the vocabulary already built in the parent note: singular values $σ_{i}$ , right singular vectors $v_{i}$ (columns of $V$ ), left singular vectors $u_{i}$ (columns of $U$ ), the shape-of- $A$ matrix $Σ$ , and the gateway object $A^{⊤} A$ . If a symbol feels unfamiliar, return to the parent derivation first.

True or false — justify

Every matrix has an SVD.

True — the derivation never assumed

A

was square, invertible, or symmetric; it only used that

A^{⊤} A

is symmetric PSD, which holds for any real

A

. See Spectral Theorem.

Singular values can be negative if

A

has negative entries.

False —

σ_{i} = λ_{i} (A^{⊤} A)

and the eigenvalues

λ_{i}

of a PSD matrix are

\geq 0

, so the square root is real and non-negative by construction, regardless of

A

's entries.

The matrix

Σ

is always square.

False —

Σ

has the shape of $A$ (

m \times n

); only when

m = n

is it square. Otherwise it carries zero-padding rows or columns.

A

is symmetric, its singular values equal its eigenvalues.

False in general, true only when

A

is also PSD. For symmetric

A

σ_{i} = ∣ λ_{i} (A) ∣

, so a negative eigenvalue

- 3

gives singular value

3

U

and

V

are uniquely determined by

A

False — repeated singular values leave freedom to rotate within their shared subspace, and each singular vector's sign can flip (if you flip

v_{i}

you must flip

u_{i}

to keep

A v_{i} = σ_{i} u_{i}

Σ

, however, is unique.

The columns of

U

live in the same space as the columns of

V

False —

u_{i} \in R^{m}

(output space),

v_{i} \in R^{n}

(input space). SVD uses two separate orthonormal frames, which is exactly what distinguishes it from eigen-decomposition.

Zero singular values contribute nothing to

A

True for reconstructing

A

(their outer products vanish), but they are not useless — the corresponding

v_{i}

span the null space of

A

, so they encode what

A

destroys.

An orthogonal matrix has all singular values equal to 1.

True — if

Q^{⊤} Q = I

then

Q^{⊤} Q

has every eigenvalue

1

, so every

σ_{i} = 1 = 1

; an orthogonal map rotates/reflects without stretching.

The number of nonzero singular values equals the rank of

A

True —

rank (A) = rank (A^{⊤} A)

= number of nonzero eigenvalues of

A^{⊤} A

= number of

σ_{i} > 0

, which the derivation calls

r

Spot the error

"To get

U

, just take the eigenvectors of

A A^{⊤}

and pair them arbitrarily with the

v_{i}

The pairing and sign are not free: eigenvectors of

A A^{⊤}

are determined only up to sign, so you might land on

- u_{i}

and break

A v_{i} = σ_{i} u_{i}

. Fix: pick

V

first, then define

u_{i} = A v_{i} / σ_{i}

u_{i} = A v_{i} / σ_{i}

defines all

m

columns of

U

It only defines

u_{i}

for

σ_{i} > 0

(i.e.

i \leq r

). The remaining

u_{r + 1}, \dots, u_{m}

must be built by extending to a full orthonormal basis, e.g. via Gram–Schmidt.

"Since

σ_{1} \geq σ_{2} \geq \dots

, the largest singular value is the largest eigenvalue of

A

σ_{1}

is the largest eigenvalue of

A^{⊤} A

, square-rooted — not an eigenvalue of

A

. For

A = (0020)

, all eigenvalues of

A

are

0

yet

σ_{1} = 2

A = U Σ V^{⊤}

A^{- 1} = V Σ^{- 1} U^{⊤}

always."

Only if

A

is square and every

σ_{i} > 0

. If any

σ_{i} = 0

then

Σ

isn't invertible; the correct tool is the pseudoinverse

A^{+} = V Σ^{+} U^{⊤}

, inverting only the nonzero

σ_{i}

"We divide

A v_{i}

σ_{i}

merely as a normalisation convenience."

It does more than normalise: it also fixes the correct sign/direction pairing so that the column identity

A V = U Σ

holds exactly. Skip it and the whole assembly step fails.

A^{⊤} A

and

A A^{⊤}

have completely different eigenvalues."

They share all nonzero eigenvalues (both equal

σ_{i}^{2}

); they differ only in how many extra zero eigenvalues pad the larger matrix. This is why one

Σ

serves both

U

and

V

Σ

diagonal means

A

acts diagonally, so

A

is diagonalizable."

Σ

is diagonal in the two SVD frames, not in the standard basis, and

U \neq = V

in general — that's a change of basis on input and output separately, which is not eigen-diagonalization. A defective (non-diagonalizable) matrix still has an SVD.

Why questions

Why do we start the derivation from

A^{⊤} A

instead of

A

itself?

A

may be rectangular with no eigenvectors at all, but

A^{⊤} A

is always symmetric PSD, so the Spectral Theorem hands us an orthonormal eigenbasis with real non-negative eigenvalues for free.

Why must the eigenvalues of

A^{⊤} A

be non-negative?

For any

x

x^{⊤} A^{⊤} A x = ∥ A x ∥^{2} \geq 0

, so

A^{⊤} A

is PSD; a PSD matrix's eigenvalues are exactly these values on eigenvectors, hence

\geq 0

, which is what lets us take real square roots.

Why does SVD need two orthogonal matrices when eigen-decomposition needs only one?

Eigen-decomposition maps a space to itself, so one basis suffices; a general

A

maps input space

R^{n}

to a different output space

R^{m}

, so it needs a source frame (

V

) and a landing frame (

U

Why are the

u_{i}

automatically orthonormal without any extra work?

Because

u_{i}^{⊤} u_{j} = \frac{1}{σ _{i} σ _{j}} v_{i}^{⊤} A^{⊤} A v_{j} = \frac{σ _{j}}{σ _{i}} δ_{ij}

, the orthonormality of the

v_{i}

(eigenvectors) transfers directly to the

u_{i}

Why is SVD the natural tool behind Principal Component Analysis (PCA)?

The right singular vectors of the centred data are the eigenvectors of the covariance matrix

\propto A^{⊤} A

, and the

σ_{i}^{2}

rank the variance captured along each direction — exactly the principal components.

Why does keeping only the top

k

singular values give the best rank-

k

approximation?

Because the

u_{i}, v_{i}

frames are orthonormal, the

σ_{i}

contributions are independent and additive; dropping the smallest ones removes the least "stretch", which the Eckart–Young theorem shows is optimal — see Low-Rank Approximation.

Why order the singular values

σ_{1} \geq σ_{2} \geq \dots

The ordering is a convention that makes

Σ

unique, groups the nonzero values first (cleanly separating rank

r

from the null part), and lets truncation always keep the most significant directions.

Edge cases

What is the SVD of the zero matrix

A = 0

(size

m \times n

All

σ_{i} = 0

Σ = 0

;

U

and

V

can be any orthogonal matrices, and

U 0 V^{⊤} = 0

holds trivially. Rank is

0

, every

v_{i}

spans the null space.

What happens to

U

when

r < m

(fewer nonzero singular values than output dimensions)?

The formula

u_{i} = A v_{i} / σ_{i}

produces only

r

vectors; the remaining

m - r

columns are chosen freely as an orthonormal basis of the left null space

null (A^{⊤})

Does SVD exist for a matrix with complex eigenvalues, like a rotation?

Yes — SVD is built from

A^{⊤} A

's real eigenvalues, never

A

's. A

9 0^{\circ}

rotation has complex eigenvalues

\pm i

but singular values

1, 1

, correctly reporting "no stretching."

What does

Σ

look like for a tall, full-column-rank matrix (

m > n

r = n

m \times n

block with the

n

positive singular values on the diagonal and

m - n

all-zero rows padded below; there are no zero singular values, so all

v_{i}

lie in the row space.

If a singular value is repeated, what freedom appears in the singular vectors?

Within the shared eigenspace of

A^{⊤} A

any orthonormal rotation of the

v_{i}

is valid (and forces a matching rotation of the

u_{i}

); the decomposition is no longer unique but

Σ

still is.

For a

1 \times 1

matrix

A = [a]

, what is its SVD?

σ_{1} = ∣ a ∣

V = [1]

, and

U = [sign (a)]

(with

U = [1]

a = 0

), so

U Σ V^{⊤} = ∣ a ∣ \cdot sign (a) = a

; the sign lives in

U

, the magnitude in

Σ

What is the null space of

A

when no singular value is zero?

It is

{0}

: every

v_{i}

has

σ_{i} > 0

A v_{i} \neq = 0

, meaning

A

injective on its domain and

r = n

; nothing gets collapsed.

Recall One-line summary of the traps

Singular values come from $A^{⊤} A$ (real, $\geq 0$ , never $A$ 's eigenvalues); $Σ$ has $A$ 's shape; $U$ and $V$ live in different spaces and are pinned together by $u_{i} = A v_{i} / σ_{i}$ ; zeros encode the null space, not garbage.