Singular Value Decomposition (SVD) — full derivation: Visual walkthrough

We build everything from the ground up. If you have never seen a matrix "act" on space, start at Step 0 and do not skip.

Step 0 — What does "a matrix acts on space" even mean?

WHAT. A matrix $A$ is a machine. You feed it an arrow (a vector $x$ ), it spits out another arrow $A x$ . Feeding it every arrow at once, we see what it does to a whole shape.

WHY start here. Every claim in SVD is a claim about this machine's shape-changing behaviour. We must be able to picture the input shape and the output shape before any symbol appears.

PICTURE. Take the unit circle — all arrows of length $1$ , pointing every direction. Watch what $A$ does to it.

Figure — Singular Value Decomposition (SVD) — full derivation

The big question the whole derivation answers:

Step 1 — The two frames we are hunting for

WHAT. We want two special sets of perpendicular arrows:

$v_{1}, v_{2}$ in the input circle — the arrows that will land exactly on the ellipse's axes.
$u_{1}, u_{2}$ in the output — unit arrows pointing along those axes.
$σ_{1}, σ_{2}$ — how long each axis is (the stretch amounts).

WHY. If we find these, then the machine is completely described by the sentence " $A$ sends $v_{i}$ to a $σ_{i}$ -times-longer arrow in direction $u_{i}$ ." Everything else is bookkeeping.

PICTURE. Same circle-to-ellipse, now with the special arrows drawn in.

The target relationship, term by term: $machine A special input v_{i} = stretch σ_{i} output direction u_{i}$

Read it aloud: " $A$ acting on $v_{i}$ equals $σ_{i}$ lots of $u_{i}$ ." Our job for the rest of the page: prove such $v_{i}, u_{i}, σ_{i}$ always exist and find them.

Step 2 — We cannot attack $A$ directly, so we build $A^{⊤} A$

WHAT. $A$ can be rectangular, tilted, without perpendicular eigen-arrows — hard to grip. So we form a friendlier matrix: $A^{⊤} A$ . Here $A^{⊤}$ (read " $A$ transpose") is $A$ flipped across its diagonal.

WHY this tool and not another. We need a matrix with guaranteed perpendicular special directions. A matrix has a full set of perpendicular eigen-arrows exactly when it is symmetric (that is the Spectral Theorem). And $A^{⊤} A$ is always symmetric — proof in the picture. That is the whole reason we detour through it.

PICTURE. Why is $A^{⊤} A$ symmetric? Flipping it does nothing.

$(A^{⊤} A)^{⊤} = A^{⊤} (A^{⊤})^{⊤} = A^{⊤} A$

Term by term: $(X Y)^{⊤} = Y^{⊤} X^{⊤}$ flips the order; $(A^{⊤})^{⊤} = A$ undoes a flip. The result equals what we started with — that is the definition of symmetric.

Step 3 — The eigen-arrows of $A^{⊤} A$ become our $v_{i}$

WHAT. Solve the eigen-problem for $A^{⊤} A$ : find perpendicular unit arrows $v_{i}$ and numbers $λ_{i}$ with $A^{⊤} A v_{i} = λ_{i} v_{i}, v_{i}^{⊤} v_{j} = δ_{ij} .$ Then define $σ_{i} := λ_{i}$ .

WHY. An eigenvector $v_{i}$ is an arrow the matrix does not rotate — it only scales it by $λ_{i}$ (see Eigenvalues and Eigenvectors). These un-rotated, mutually perpendicular arrows are exactly the special input directions we hunted in Step 1. The symbol $δ_{ij}$ (the Kronecker delta) is just shorthand: $1$ when $i = j$ , $0$ otherwise — it says "perpendicular unit arrows".

PICTURE. $A^{⊤} A$ leaves each $v_{i}$ pointing the same way, only scaled.

Order the arrows so $σ_{1} \geq σ_{2} \geq \dots \geq σ_{r} > 0 = σ_{r + 1} = \dots$ . Here $r$ is the rank: the count of non-zero stretches. Stacking the arrows side by side gives the matrix $V = [v_{1} ∣ \dots ∣ v_{n}]$ , which is orthogonal ( $V^{⊤} V = I$ ) because its columns are perpendicular unit arrows.

Step 4 — Push each $v_{i}$ through $A$ to get $u_{i}$

WHAT. For each $i$ with $σ_{i} > 0$ , define $u_{i} := \frac{1}{σ _{i}} A v_{i} .$

WHY divide by $σ_{i}$ ? $A v_{i}$ points the right way (along an ellipse axis) but has the wrong length. How long is it? Compute: $∥ A v_{i} ∥^{2} = (A v_{i})^{⊤} (A v_{i}) = v_{i}^{⊤} = σ_{i}^{2} v_{i} A^{⊤} A v_{i} = σ_{i}^{2} v_{i}^{⊤} v_{i} = σ_{i}^{2} .$ Term by term: regroup the arrows; substitute the eigen-equation from Step 3; use $v_{i}^{⊤} v_{i} = 1$ . So $∥ A v_{i} ∥ = σ_{i}$ . Dividing by $σ_{i}$ rescales it to length $1$ — now it is a legit unit direction $u_{i}$ . This is also where $λ_{i}$ pays off: $∥ A v_{i} ∥ = σ_{i} = λ_{i}$ , one clean stretch.

PICTURE. Watch $v_{i}$ get sent to $A v_{i}$ (length $σ_{i}$ ), then shrunk to the unit arrow $u_{i}$ .

Are the $u_{i}$ perpendicular to each other? Yes — check for $i \neq = j$ : $u_{i}^{⊤} u_{j} = \frac{1}{σ _{i} σ _{j}} (A v_{i})^{⊤} (A v_{j}) = \frac{1}{σ _{i} σ _{j}} v_{i}^{⊤} A^{⊤} A v_{j} = \frac{σ _{j}^{2}}{σ _{i} σ _{j}} v_{i}^{⊤} v_{j} = 0.$ The last equality holds because $v_{i}^{⊤} v_{j} = 0$ (Step 3's perpendicularity). So $u_{1}, \dots, u_{r}$ are perpendicular unit arrows.

Step 5 — Complete the output basis (the leftover directions)

WHAT. We have $u_{1}, \dots, u_{r}$ (one per non-zero $σ_{i}$ ). If the output space $R^{m}$ is bigger than $r$ , we still need $u_{r + 1}, \dots, u_{m}$ to fill it. We invent them using the Gram-Schmidt Process: pick any arrows filling the gaps, then straighten them to be perpendicular unit arrows.

WHY. $U$ must be a full $m \times m$ orthogonal matrix (a complete frame), even if $A$ only reaches an $r$ -dimensional slice of the output. The extra $u$ 's point into directions $A$ never lands on — the left null space.

PICTURE. Two reached axes $u_{1}, u_{2}$ plus one unreached direction $u_{3}$ completing 3D space.

Stacking all of them gives $U = [u_{1} ∣ \dots ∣ u_{m}]$ , orthogonal by construction. Together with Step 3's $V$ , these are the four bases of the Four Fundamental Subspaces.

Step 6 — Assemble $A = U Σ V^{⊤}$

WHAT. Collect the single fact $A v_{i} = σ_{i} u_{i}$ (true for every $i$ : when $σ_{i} = 0$ both sides are the zero arrow) into one matrix equation.

WHY. Doing the same thing to every column at once is matrix multiplication. Stacking $A v_{i} = σ_{i} u_{i}$ as columns: $A [v_{1} \dots v_{n}] = [σ_{1} u_{1} \dots σ_{n} u_{n}] ⟹ A V = U Σ.$ Here $Σ$ is the $m \times n$ box with the $σ_{i}$ on its diagonal (same shape as $A$ — it is not square in general), so that $U Σ$ places $σ_{i}$ in front of $u_{i}$ .

Right-multiply both sides by $V^{⊤}$ . Since $V$ is orthogonal, $V V^{⊤} = I$ (rotating then un-rotating changes nothing): $A V V^{⊤} = U Σ V^{⊤} ⟹ A = U Σ V^{⊤} .$

PICTURE. The famous three-move factory: input frame → turn ( $V^{⊤}$ ) → stretch ( $Σ$ ) → turn ( $U$ ) → output.

Read the moves right to left (the order they hit an input $x$ in $U Σ V^{⊤} x$ ):

$V^{⊤}$ : rotate the input so the special arrows $v_{i}$ align with the coordinate axes.
$Σ$ : stretch axis $i$ by $σ_{i}$ (and squash to $0$ any axis with $σ_{i} = 0$ ).
$U$ : rotate the stretched result into its final resting frame.

Step 7 — The degenerate cases (nothing is left to chance)

WHAT & WHY. SVD claims to work for every matrix. So we must survive the ugly inputs. Each gets a panel below.

The one-picture summary

Everything on this page in a single diagram: perpendicular input frame $v_{i}$ → each stretched by $σ_{i}$ → landing on the perpendicular output frame $u_{i}$ , tracing the ellipse. The two frames plus the stretch numbers are the SVD.

Recall Feynman retelling — say it in plain words

A matrix is a machine that turns the unit circle into an ellipse. An ellipse has a long axis and a short axis — the only special directions in sight. I want to know which input arrows fed those axes and by how much they grew. The raw matrix is too messy to ask directly, so I build $A^{⊤} A$ , which is guaranteed to have a perpendicular set of un-rotated arrows $v_{i}$ (that is what symmetry buys me). Their scaling numbers are $λ_{i} \geq 0$ ; I take square roots to get the true stretches $σ_{i}$ . Pushing each $v_{i}$ through $A$ and rescaling to length one gives the output axes $u_{i}$ — and because I define them this way, their signs and pairing come out right automatically. I pad out any missing output directions with Gram–Schmidt. Assembling the single fact " $A$ sends $v_{i}$ to $σ_{i} u_{i}$ " column by column, and undoing one rotation, gives $A = U Σ V^{⊤}$ : turn, stretch, turn. Zero stretches just mean some input arrows got crushed to a point — that is the null space, and the theorem swallows it without complaint.

Recall

Why detour through $A^{⊤} A$ instead of $A$ ? ::: Because $A^{⊤} A$ is always symmetric and PSD, so the Spectral Theorem guarantees perpendicular eigen-arrows and non-negative eigenvalues — $A$ itself offers no such guarantee. What single fact, stacked as columns, gives the whole SVD? ::: $A v_{i} = σ_{i} u_{i}$ for all $i$ , which stacks to $A V = U Σ$ , hence $A = U Σ V^{⊤}$ . What geometric object is the unit circle mapped to, and what are its axes? ::: An ellipse; its semi-axis lengths are the singular values $σ_{i}$ and its axis directions are the left singular vectors $u_{i}$ .