Pseudoinverse: Visual walkthrough — Maths deep dive

Step 1 — What does a matrix even do to space?

WHAT. We draw the input world (left) and the output world (right). The machine $A$ carries the arrow $x$ across.

WHY. Before "inverting" anything, we must picture what forward motion looks like. An inverse is just "walk back along this arrow." If forward motion loses information, backward motion cannot be perfect — and that loss is the whole reason $A^{+}$ exists.

PICTURE. Look at the figure. The input vector $x$ (lavender) is squeezed and rotated into $A x$ (coral) living on the tilted column-space sheet. Not every point in the output world lies on that sheet — those unreachable points are exactly where trouble begins.

In symbols, reading each piece:

$the machine A input, n numbers x = output, lands on column space A x$

Step 2 — The problem: $b$ is off the sheet

WHAT. We place $b$ (coral dot) floating above the tilted sheet, and mark the closest point on the sheet as $\hat{b}$ .

WHY this idea and not another. "Closest" is the only fair meaning of best when exact is impossible. Closest means shortest distance — and shortest distance from a point to a flat sheet is always measured along the perpendicular (any slanted path is longer). That perpendicular is the seed of everything.

PICTURE. The dashed coral line from $b$ down to $\hat{b}$ meets the sheet at a right angle (the small square). The length of that dashed line is the error we cannot avoid — the residual.

$squared length of the error arrow ∥ A x - b ∥^{2} ⟶ make this as small as possible$

We square the length only because squaring removes the awkward square-root and gives a smooth bowl-shaped function that is easy to minimise — the minimum sits at the same place either way.

Step 3 — The perpendicular condition (this is orthogonal projection)

WHAT. We say " $A x - b$ is perpendicular to all columns of $A$ " in one clean line.

WHY the dot product / transpose. To test perpendicularity against every column at once, we hit the residual with $A^{⊤}$ (each row of $A^{⊤}$ is a column of $A$ ). Getting all zeros means "no leftover leans along the sheet." This is the exact algebraic twin of the geometric right angle in Step 2 — see Orthogonal Projection.

PICTURE. The residual arrow (coral) stands up off the mint sheet; the columns $a_{1}, a_{2}$ lie flat in it. Every angle between the residual and a column is $9 0^{\circ}$ .

$rows = the columns of A A^{⊤} residual arrow (A x - b) = no leftover along the sheet 0$

Expanding the bracket gives the normal equations — "normal" is old-fashioned for "perpendicular":

$small n \times n machine A^{⊤} A x = shadow of b on the sheet A^{⊤} b$

Step 4 — Tall case: solve it, and $A^{+}$ is born

WHAT. We isolate $x$ by undoing $A^{⊤} A$ . Whatever now multiplies $b$ is the pseudoinverse — that is literally its definition here.

WHY this is allowed. Independent columns are exactly the condition that $A^{⊤} A$ has no zero direction, so its ordinary inverse exists. If columns were dependent, $A^{⊤} A$ would flatten a direction to zero and refuse to invert — that is Step 6.

PICTURE. $A^{+}$ (mint arrow) travels from the floating $b$ back into the input world, landing on the least-squares answer $\overset{x}{^}$ . It is a genuine left inverse: doing $A$ then $A^{+}$ returns you exactly, since

$A^{+} A = (A^{⊤} A)^{- 1} cancels A^{⊤} A = I_{n} .$

This is precisely Least Squares Regression: the best-fit line lives here.

Step 5 — Fat case: too many answers, pick the shortest

WHAT. Among all valid $x$ , we single out the one with the smallest length $∥ x ∥$ .

WHY "shortest". All solutions differ only by vectors in the null space (directions $A$ crushes to zero — free, wasted energy; see Four Fundamental Subspaces). Removing all null-space content leaves the unique piece lying in the row space. The trick $x = A^{⊤} w$ forces $x$ into the row space automatically, killing the junk. Shortest = perpendicular to the solution line.

PICTURE. The green solution line drifts through space; the mint dot is the point on it nearest the origin, sitting perpendicular to the line's direction (the null-space arrow).

Here $A^{+}$ is a right inverse: $A A^{+} = A A^{⊤} (A A^{⊤})^{- 1} = I_{m}$ .

Step 6 — The degenerate case: a squashed direction

WHAT. We picture the machine flattening a whole axis to zero — a pancake.

WHY we cannot fake it. Someone always suggests "just divide by a tiny $ϵ$ instead of zero." But $1/ ϵ$ explodes, blasting noise sky-high (parent Mistake 3). The truthful move: a crushed direction carries no information, so its inverse contribution is exactly zero.

PICTURE. The input square (lavender) is pressed flat onto a single line (coral). The vertical direction is gone forever — no honest arrow brings it back.

Step 7 — SVD: the universal machine, direction by direction

WHAT. Because $U, V$ are pure rotations, undoing them is just rotating back ( $U^{⊤}$ , $V$ ). Only the stretch step needs care.

WHY invert nonzeros only. In the diagonal $Σ$ , a nonzero stretch $σ_{i}$ is reversed by shrinking with $1/ σ_{i}$ . A zero stretch (a crushed direction from Step 6) has no reverse — you leave it as a zero. Read $Σ^{+}$ term by term:

$σ_{i} alive \frac{1}{σ _{i}}, 0 dead 0.$

PICTURE. Follow the three coloured stages left-to-right for $A$ : rotate, stretch, rotate. Then the return path $A^{+}$ runs right-to-left: unrotate, un-stretch (only live axes), unrotate. The dead axis stays flat both ways.

This single formula reproduces Step 4 and Step 5 as special cases, and survives Step 6 unscathed. The squared stretches $σ_{i}^{2}$ are the eigenvalues of $A^{⊤} A$ .

The one-picture summary

The whole journey on one canvas: forward through $A$ we rotate, stretch, rotate and may lose a direction; backward through $A^{+}$ we unrotate, un-stretch the live parts, unrotate — recovering everything recoverable and nothing more.

Recall Feynman retelling — the whole walkthrough in plain words

A matrix is a machine that grabs your arrow and reshapes it, dropping it onto a flat sheet. You then ask it to hand your arrow back.

If your target floats off the sheet (noisy data), a perfect return is impossible — so $A^{+}$ drops a straight-down perpendicular to the nearest sheet point and returns that. That perpendicular is least squares (Steps 2–4), giving $A^{+} = (A^{⊤} A)^{- 1} A^{⊤}$ for tall matrices.

If instead the machine allows many correct returns (wide matrix), $A^{+}$ picks the shortest, laziest one — no wasted null-space wandering (Step 5), giving $A^{+} = A^{⊤} (A A^{⊤})^{- 1}$ .

And when the machine flattens a direction to nothing (Step 6), no honest arrow can un-flatten it — so $A^{+}$ leaves it at zero. The SVD (Step 7) does this automatically: undo the rotations, invert only the living stretches, skip the dead ones. That is $A^{+} = V Σ^{+} U^{⊤}$ — the one formula to rule them all.

Recall

The residual at the least-squares solution is perpendicular to what? ::: The column space of $A$ (every column of $A$ ). Why is $A^{⊤} A$ invertible in the tall case but not the rank-deficient case? ::: Independent columns give no zero direction; rank deficiency leaves a crushed direction that $A^{⊤} A$ maps to zero. In the SVD formula, what happens to a zero singular value? ::: It stays zero in $Σ^{+}$ — that direction carries no information to invert. What makes the fat-matrix answer unique among infinitely many solutions? ::: It is the shortest one, perpendicular to the null space (all null content removed).

Connections

Pseudoinverse — parent note; this page is its visual derivation.
Orthogonal Projection — Step 2–3, the perpendicular that defines "closest."
Least Squares Regression — Step 4, the tall-matrix application.
Singular Value Decomposition — Step 7, the universal engine.
Four Fundamental Subspaces — column space, row space, null space throughout.
Eigenvalues and Eigenvectors — $σ_{i}^{2}$ as eigenvalues of $A^{⊤} A$ .
Matrix Inverse — the special square case where all this collapses to $A^{- 1}$ .