Intuition The one core ideaAny matrix, no matter how ugly, is secretly just rotate → stretch → rotate . To read the sentence "A = U Σ V ⊤ rotations (U V pure stretch (Σ
This is the prerequisite page for Singular Value Decomposition (SVD) — full derivation . Read it first. We will name every symbol the parent note fires at you, draw the picture behind it, and explain why the derivation cannot proceed without it .
Throughout, watch for two families of special vectors: v 1 , … , v n right singular vectors , columns of V u 1 , … , u m left singular vectors , columns of U "the special input/output directions SVD will produce."
Definition Matrix = a machine that moves arrowsA matrix A function that eats a vector and spits out a vector . Feed it an arrow x A x
Plain words: rows and columns of numbers.The picture: a transformation of space. A grid of arrows before; the same arrows moved after.Why the topic needs it: SVD is a claim about what this machine does to space — it says the messy motion is always "spin, stretch, spin". So we must first see a matrix as motion, not as a spreadsheet.
Figure s01 (below): the same grid of arrows shown twice — untouched on the left, and after the machine A A motion of space , not a spreadsheet.
Shape notation. We write A ∈ R m × n
A ∈ R m × n
R real numbers (ordinary decimals like 3 , − 1.5 , 2 m × n m n The picture: the machine takes arrows living in n m n m Why: SVD's whole point is that m n different rooms , so we need two sets of directions, one per room.
A vector x x = ( x 1 , x 2 , … , x n ) ⊤ ⊤ transpose ) here just means "stand this row up as a column".
The picture: an arrow in space with a length and a direction.Why the topic needs it: SVD is entirely a story about what A
A ⊤ transpose flips a matrix across its diagonal: row i i A m × n A ⊤ n × m
The picture: hold the grid up to a mirror placed along the top-left-to-bottom-right diagonal.Why the topic needs it: the entire derivation is built on the object A ⊤ A
A ⊤ A
Definition Dot product and lengthThe dot product of two vectors is x ⊤ y = x 1 y 1 + x 2 y 2 + ⋯ + x n y n squared length :
∥ x ∥ 2 = x ⊤ x = x 1 2 + ⋯ + x n 2 . ∥ x ∥ length (norm) of the arrow — straight-line distance from origin to tip, by Pythagoras.
Why the topic needs it: the proof line ∥ A v ∥ 2 = ( A v ) ⊤ ( A v ) = v ⊤ A ⊤ A v nothing but the dot-product-is-squared-length trick. Each singular value σ ∥ A v ∥ v
Figure s02 (below): an arrow x = ( 3 , 2 ) ∥ x ∥ x x ⊤ x = x 1 2 + x 2 2 is Pythagoras — the squared length.
Intuition Why the dot product measures angle toox ⊤ y = ∥ x ∥ ∥ y ∥ cos θ θ
If x ⊤ y = 0 perpendicular (cos 90° = 0
x ⊤ x = ∥ x ∥ 2 0 cos 0 = 1
Definition Orthonormal setA set of vectors is orthonormal if every vector has length 1 normal ) and every pair is perpendicular (ortho ). Compactly:
v i ⊤ v j = δ ij .
The picture: a set of unit arrows all at right angles — a clean, un-skewed set of axes.Why the topic needs it: the two frames SVD produces (the v u U V rotations rather than sloppy skews.
Definition Identity matrixI I I x = x 1 0 I n n × n
Definition Orthogonal matrixA square matrix Q orthogonal if its columns form an orthonormal set, equivalently Q ⊤ Q = I
The picture: Q rigid motion — a rotation or a reflection. It never stretches, never squishes; lengths and angles are preserved. Feed it a square grid, get a rotated square grid of the same size.Why the topic needs it: U V A = U Σ V ⊤ exactly these. V U
Figure s03 (below): the unit axes before (left) and after an orthogonal matrix Q rotate . This is the picture of U V
Q ⊤ Q = I i j ( i , j ) Q ⊤ Q Q ⊤ Q = I δ ii = 1 δ ij = 0 Q ⊤ Q = I Q ⊤ = Q − 1 undoing the rotation is free.
First, the word we are about to lean on:
r σ r rank r A genuinely independent output directions the machine can produce — the dimension of its output image. Full picture and "why" in §4; for now just read r
σ 1 ≥ σ 2 ≥ ⋯ ≥ σ r > 0 = σ r + 1 = …
Here σ r nonzero stretch — the first r r
The span of a set of vectors is every arrow you can build by scaling and adding them. Two non-parallel arrows in a plane span the whole plane; two parallel ones span only a line.
The picture: the full region "reachable" by combining the arrows — a line, a plane, or all of space.Why the topic needs it: column space, row space and null space are all spans of chosen vectors; SVD's payoff is that it hands you clean orthonormal spanning sets for each.
r A
The picture: feed the machine the whole input space; the shadow it casts in output space is r Why: the parent splits singular values into σ 1 … σ r > 0 σ r + 1 ⋯ = 0
The Four Fundamental Subspaces are the four rooms every matrix lives in. SVD hands you an orthonormal basis for all four at once — that is why we name them here.
A A rows (equivalently the column space of A ⊤ not crushed.
The picture: the r Why the topic needs it: the right singular vectors v 1 , … , v r σ i > 0
We can now define precisely what was promised at the top.
Definition Right singular vectorsv i V right singular vectors v 1 , … , v n input space R n V = [ v 1 ∣ ⋯ ∣ v n ]
The picture: the clean input frame SVD chooses — the axes along which A Why the topic needs it: V ⊤ first rotation in "rotate → stretch → rotate"; it spins your input onto this special frame.
Definition Left singular vectorsu i U left singular vectors u 1 , … , u m output space R m U = [ u 1 ∣ ⋯ ∣ u m ]
The picture: the clean output frame SVD lands in — A v i σ i u i Why the topic needs it: U second rotation; after stretching by Σ
Definition Eigenvector & eigenvalueAn eigenvector v M direction the machine does not change — it only scales it: M v = λ v λ eigenvalue .
The picture: most arrows get knocked off their line by M Why: the derivation's engine is the eigenproblem A ⊤ A v i = σ i 2 v i A ⊤ A are the right singular vectors; the eigenvalues are the squared singular values.
Full detail lives in Eigenvalues and Eigenvectors .
Definition Symmetric matrixM symmetric if M ⊤ = M A ⊤ A ( A ⊤ A ) ⊤ = A ⊤ A
Definition Positive semi-definite (PSD)M PSD if x ⊤ M x ≥ 0 x M = A ⊤ A x ⊤ A ⊤ A x = ∥ A x ∥ 2 ≥ 0
To extend the orthonormal u i Gram-Schmidt Process — the standard machine for turning any spanning set into an orthonormal one.
Symbol
Says out loud
Picture
Role in SVD
A the matrix
motion of space
the thing we decompose
A ⊤ transpose
mirror across diagonal
builds A ⊤ A
x ⊤ y dot product
length & angle
measures σ i = ∥ A v i ∥
δ ij Kronecker delta
identity grid
encodes orthonormality
U , V orthogonal
rotations
the two spins (left/right singular vectors)
Σ diagonal
pure stretch
the singular values
λ i , v i eigen-pair
invariant direction
σ i 2
r rank
output dimension
count of nonzero σ i
Orthonormal sets and delta
Orthogonal matrices U and V
Symmetric matrix A-transpose-A
Eigenvalues and eigenvectors
SVD A = U Sigma V-transpose
Test yourself — reveal only after answering.
I can read A ∈ R m × n Input is R n n R m m
I can compute a transpose A ⊤ Flip across the main diagonal: row i i m × n n × m
I can write squared length as a dot product ∥ x ∥ 2 = x ⊤ x = x 1 2 + ⋯ + x n 2
I know what x ⊤ y = 0 The two arrows are perpendicular.
I can state what δ ij 1 i = j 0
I can define an orthogonal matrix two ways Columns are orthonormal; equivalently Q ⊤ Q = I Q ⊤ = Q − 1
I can describe what a diagonal matrix does to space Stretches each coordinate axis independently, no rotation.
I can write the eigenvalue equation M v = λ v λ
I know why A ⊤ A ( A ⊤ A ) ⊤ = A ⊤ A x ⊤ A ⊤ A x = ∥ A x ∥ 2 ≥ 0
I know what the (real) Spectral Theorem grants me A real symmetric matrix has an orthonormal eigenbasis with real eigenvalues.
I can define rank and the null space Rank = dimension of the output image; null space = inputs sent to 0
I can name the four fundamental subspaces Row space and null space (input side), column space and left null space (output side).
Ready? Proceed to Singular Value Decomposition (SVD) — full derivation . Related destinations once you understand SVD: Low-Rank Approximation , Principal Component Analysis (PCA) , Moore-Penrose Pseudoinverse .
