NumPy linear algebra — np.linalg.solve, eig, svd, norm, det: Visual walkthrough

Prerequisites you may want open: NumPy arrays and broadcasting, Vector norms and metric spaces, Eigenvalues and diagonalization.

Step 1 — What a matrix does to an arrow

WHAT. A vector is just an arrow from the origin to a point. A number pair $x = (x_{1} x_{2})$ means "walk $x_{1}$ right, $x_{2}$ up". A matrix $A$ is a machine: feed it an arrow, it hands you a new arrow $A x$ .

WHY start here. Every later symbol ( $U$ , $Σ$ , $V$ ) is also just a machine acting on arrows. If we don't feel this first, the factorization is meaningless letters.

PICTURE. Look at the blue input arrow and the red output arrow. The machine below is $A = (20 1 1.5) .$ Here each entry has a job: the top-left $2$ says "the first output coordinate takes 2 copies of $x_{1}$ "; the top-right $1$ mixes in $x_{2}$ ; the bottom row builds the second output coordinate. The arrow got both longer and tilted.

Figure — NumPy linear algebra — np.linalg.solve, eig, svd, norm, det

Step 2 — Watch the whole unit circle, not one arrow

WHAT. Instead of one arrow, feed the machine every unit-length arrow at once — the whole unit circle (all arrows of length 1 pointing every direction).

WHY. One arrow tells you almost nothing. The set of all outputs reveals the machine's personality: a circle always maps to an ellipse (a squashed, tilted circle). That ellipse is the entire secret of SVD.

PICTURE. The blue circle (inputs) becomes the red ellipse (outputs). Notice the ellipse has a longest radius and a shortest radius — and they sit at right angles to each other.

Step 3 — Name the ellipse's axes: the singular values $σ$

WHAT. The red ellipse has a major axis (its longest half-width) and a minor axis (its shortest half-width). Call their lengths $σ_{1}$ (bigger) and $σ_{2}$ (smaller). These are the singular values.

WHY these two numbers? They are the actual stretch amounts the machine applies. $σ_{1}$ is the most any unit arrow can be stretched; $σ_{2}$ is the least. Every other output length lives between them. That is why we always order them $σ_{1} \geq σ_{2} \geq 0$ .

PICTURE. Yellow arrows mark the two ellipse axes; their lengths are labelled $σ_{1}$ and $σ_{2}$ .

Step 4 — Name the input directions that hit those axes: $v_{1}, v_{2}$

WHAT. Which input arrows on the blue circle got stretched into the major and minor axes? Call them $v_{1}$ (maps to the long axis) and $v_{2}$ (maps to the short axis). These are the right singular vectors.

WHY they matter. They are the special input directions the machine treats cleanly — pure stretch, no confusion. Crucially, $v_{1}$ and $v_{2}$ come out perpendicular to each other on the input circle. Perpendicular in, perpendicular out — that orthogonality is the backbone of the whole factorization.

PICTURE. Green arrows $v_{1}, v_{2}$ sit at 90° on the blue circle; the machine sends each straight onto its matching yellow ellipse axis.

Step 5 — Name the output directions: $u_{1}, u_{2}$

WHAT. Strip the length off the ellipse axes and keep only their directions as unit arrows: $u_{1}$ (along the major axis) and $u_{2}$ (along the minor). These are the left singular vectors.

WHY. We split each output into "which way it points" ( $u_{i}$ ) times "how far it stretched" ( $σ_{i}$ ). Like the $v$ 's, the $u$ 's are perpendicular to each other. Now we have four clean ingredients: perpendicular inputs $v_{1}, v_{2}$ , perpendicular output directions $u_{1}, u_{2}$ , and stretch amounts $σ_{1}, σ_{2}$ .

PICTURE. Purple/green unit arrows $u_{1}, u_{2}$ at 90° on the output side, each equal to its yellow axis divided by its $σ$ .

Step 6 — Stack the vectors into matrices → the factorization appears

WHAT. Put the input vectors as columns of a matrix $V = [v_{1} v_{2}]$ , the output directions as columns of $U = [u_{1} u_{2}]$ , and the stretches on the diagonal of $Σ = (σ_{1} 0 0 σ_{2})$ .

WHY. The single equation $A v_{i} = σ_{i} u_{i}$ , written for both $i$ at once, becomes one matrix equation: $A V = U Σ.$ Reading it: $A V$ applies $A$ to both input columns; $U Σ$ says each result is the output direction times its stretch. Now the payoff — because $V$ 's columns are perpendicular unit vectors, $V$ is orthogonal, meaning $V^{⊤} V = I$ , so $V^{- 1} = V^{⊤}$ . Multiply both sides on the right by $V^{⊤}$ : $A = U Σ V^{⊤} .$

PICTURE. The blocks assembled: $U$ (rotate), $Σ$ (stretch), $V^{⊤}$ (rotate) side by side, with the arrow flowing through them.

Step 7 — Where the eigenvalue link hides: $A^{⊤} A$

WHAT. Multiply $A$ by its transpose $A^{⊤}$ and simplify using the factorization.

WHY. This explains the parent's claim that $σ_{i}^{2}$ are eigenvalues of $A^{⊤} A$ . Substitute and watch the rotations cancel: $A^{⊤} A = (U Σ V^{⊤})^{⊤} (U Σ V^{⊤}) = V Σ^{⊤} = I U^{⊤} U Σ V^{⊤} = V Σ^{2} V^{⊤} .$ Term by term: $U^{⊤} U = I$ because $U$ is orthogonal (its perpendicular unit columns), so it vanishes; $Σ^{⊤} Σ = Σ^{2}$ has entries $σ_{i}^{2}$ . The remaining shape $V Σ^{2} V^{⊤}$ is exactly a diagonalization (see Eigenvalues and diagonalization), so the diagonal entries $σ_{i}^{2}$ are the eigenvalues and the columns of $V$ are the eigenvectors.

PICTURE. The cancellation shown as two rotations meeting and annihilating, leaving pure stretch-squared.

Step 8 — The degenerate case: a squashed (singular) matrix

WHAT. What if the machine flattens the circle onto a line? Then the ellipse has zero minor axis: $σ_{2} = 0$ .

WHY show this. SVD must survive every input, including matrices you can't invert. When $σ_{2} = 0$ : the output is one-dimensional, $det (A) = σ_{1} σ_{2} = 0$ (matches the parent's "collapsed volume"), and $A$ is singular. np.linalg.solve would fail, but SVD keeps working — it simply reports a zero singular value. This is why SVD is the tool for Least squares regression and Principal Component Analysis (PCA): it stays sane where inversion dies.

PICTURE. The blue circle collapses to a red segment; $σ_{1}$ is the segment's half-length, $σ_{2} = 0$ drawn as a flattened axis.

The one-picture summary

Everything in a single frame: the input circle with perpendicular directions $v_{1}, v_{2}$ , flowing through rotate ( $V^{⊤}$ ) → stretch ( $Σ$ ) → rotate ( $U$ ), landing as the ellipse with axes $σ_{1} u_{1}$ and $σ_{2} u_{2}$ .

Worked example Verify the whole story in NumPy

import numpy as np
A = np.array([[2., 1.],
              [0., 1.5]])
U, s, Vt = np.linalg.svd(A)      # s = [sigma_1, sigma_2], sorted big->small
# 1) reconstruction: rotate-stretch-rotate rebuilds A
Sig = np.diag(s)
print(np.allclose(U @ Sig @ Vt, A))            # True
# 2) sigma_i^2 are eigenvalues of A^T A
lam = np.sort(np.linalg.eigvals(A.T @ A))[::-1]
print(np.allclose(s**2, lam.real))             # True
# 3) det = product of singular values (up to sign)
print(np.allclose(abs(np.linalg.det(A)), s.prod()))  # True

Recall Feynman retelling — the whole walkthrough in plain words

Imagine a stretchy machine for arrows. Feed it every arrow of length 1 — a perfect circle goes in. What comes out is always a tilted, squashed circle: an ellipse. That ellipse has a longest radius and a shortest radius, at right angles; those lengths are the singular values $σ_{1}, σ_{2}$ — the biggest and smallest stretch the machine can do. Two special input arrows (also at right angles) are the ones that landed exactly on those radii; call them $v_{1}, v_{2}$ . The directions the radii point are $u_{1}, u_{2}$ . So the machine really does just three simple things: spin the input so the special arrows line up with the axes ( $V^{⊤}$ ), stretch each axis by its $σ$ ( $Σ$ ), then spin the result into place ( $U$ ). Stack these arrows into matrices and you literally get $A = U Σ V^{⊤}$ . If the machine flattens the circle to a line, one $σ$ is zero — the matrix is singular, and SVD calmly tells you so instead of crashing.

Self-check

Circle in, what shape always comes out under a matrix?

An ellipse (squashed, tilted circle) — lines never bend under a linear map.

What geometric thing are the singular values

σ_{1}, σ_{2}

The lengths of the major and minor half-axes of the output ellipse — the max and min stretch of a unit arrow.

Why can we write

A = U Σ V^{⊤}

(not

V^{- 1}

V

's columns are perpendicular unit vectors, so

V

is orthogonal and

V^{- 1} = V^{⊤}

What does

σ_{2} = 0

tell you about

A

The circle collapsed to a line —

A

is singular,

det (A) = 0

, and its rank dropped.

How are singular values linked to eigenvalues?

σ_{i} = λ_{i} (A^{⊤} A)

; the

U^{⊤} U = I

cancellation leaves

A^{⊤} A = V Σ^{2} V^{⊤}