Foundations — Pseudoinverse

This page assumes nothing. Before you meet the pseudoinverse on the parent page, we build every symbol it uses — one brick at a time, each brick resting on the one before. We will hold off writing the pseudoinverse's own symbol until §6, once the plain inverse it copies is on the table.

1. What a matrix even is (a machine that moves arrows)

We measure a matrix by its shape: $A\in {\mathbb{R}}^{m\times n}$ means "$m$ rows, $n$ columns, all real numbers". Read it as out-size $\times$ in-size: it eats an $n$-number arrow and produces an $m$-number arrow.

Figure s01 — the machine picture. The violet arrow is your input $x$; the machine $A$ sends it to the magenta output arrow $b=Ax$; the dashed orange arrow shows the move $A$ performed. Keep this picture in mind: everything on this page is about how (and whether) that move can be reversed.

2. Columns, and where the output can land (column space)

Here is the single most useful way to read $Ax=b$. If $A$ has columns $c_1,c_2,\dots ,c_n$ (each column is itself an arrow), then

What this says: the output $b$ is a weighted sum of the columns, and the numbers in $x$ are the weights. So $A$ can only ever produce arrows you can build by mixing its columns.

Figure s02 — the reachable sheet. The magenta and violet arrows are the two columns of $A$. The pale orange cloud is every arrow you can build by mixing them: that cloud is the column space. The lone navy dot marked $b$ sits off the sheet — no mix of columns reaches it, so $Ax=b$ has no exact solution.

3. Length of an arrow, and the norm symbol $\Vert \cdot \Vert$

To say "closest" we need to measure distance. For an arrow $v=\left(\begin{array}{c}{v}_1\\ v_2\end{array}\right)$:

The symbol $\Vert v{\Vert }^2$ (length squared) drops the square root: $\Vert v{\Vert }^2=v_1^2+v_2^2$. We prefer the squared version because it has no ugly root, and "smallest length" and "smallest length-squared" happen at the same place.

4. The dot product and right angles ($v^{\top }w$)

Stacking a row $v^{\top }$ against a column $w$ multiplies matching entries and adds them — this is the dot product:

Figure s03 — the right-angle test. The magenta arrow $v$ and violet arrow $w$ meet at a corner marked by the little orange square (the standard "right angle" symbol). Their dot product is exactly $0$. Read the picture as: dot product zero $\iff$ perpendicular.

5. Projection: the shadow onto the sheet

Put these together and you get the star player.

On the parent page you will read "$A{A}^{+}$ is the orthogonal projection onto the column space" (the symbol $A^{+}$ is named in §6 below). It means: feeding $b$ through $A$ then through the fair-undo machine doesn't return $b$ itself; it returns the shadow of $b$ on the reachable sheet. See Orthogonal Projection for the full picture.

6. The plain inverse $A^{-1}$, and its fair sibling $A^{+}$

$A^{-1}$ fails to exist in two situations:

7. The nullspace and row space — why many inputs can give the same output

When the machine "squishes", several different input arrows collapse onto the same output. The directions that get squished flat to zero have a name.

Why this creates many solutions. Suppose $A{x}_0=b$ solves your problem. Take any nullspace arrow $n$ (so $An=0$). Then $A(x_0+n)=A{x}_0+An=b+0=b,$ so $x_0+n$ also solves it. A whole family of solutions, one per nullspace arrow! The pseudoinverse's job in this case is to pick the one shortest member of that family — see §3's minimum-norm idea.

Figure s04 — squish and shortest choice. The machine flattens the whole plane onto the magenta line (its column space); the violet dashed line is the nullspace (arrows squished to zero). All the navy dots are inputs giving the same output; the filled orange dot is the shortest one — it sits perpendicular to the nullspace, exactly the minimum-norm answer $A^{+}$ returns.

8. Rank — how much the machine flattens

Rank labels the three cases the parent solves:

• Full column rank (rank $=n$, tall & thin): columns all independent — nothing wasted going in, and the nullspace is just the zero arrow.
• Full row rank (rank $=m$, short & fat): output can reach everything — many inputs do it, so a nontrivial nullspace forces the minimum-norm choice.
• Rank-deficient: some directions collapse to zero; only the Singular Value Decomposition survives.

9. Singular values ${\sigma }_i$ — the stretch factors

A ${\sigma }_i=0$ means one direction of the ellipse got squashed flat to a point — information destroyed, and that squashed direction is exactly a nullspace direction from §7. You can undo a stretch by ${\sigma }_i$ by dividing by ${\sigma }_i$… unless ${\sigma }_i=0$, because you cannot un-squash a point back into a line (there's no $1/0$). That is the whole reason the SVD pseudoinverse "leaves zeros as zeros". The ${\sigma }_i^2$ are the eigenvalues of $A^{\top }A$ — see Eigenvalues and Eigenvectors.

Prerequisite map

Equipment checklist

Test yourself below. Each item is written as prompt ::: answer — read only the left side, say your answer aloud, then check against the right.

Connections

• Pseudoinverse — the topic these foundations build toward.
• Singular Value Decomposition — where singular values come from.
• Orthogonal Projection — the "closest shadow" made precise.
• Matrix Inverse — the perfect undo that $A^{+}$ generalizes.
• Four Fundamental Subspaces — column space, row space, nullspace and left nullspace.
• Eigenvalues and Eigenvectors — ${\sigma }_i^2$ are eigenvalues of $A^{\top }A$.
• Least Squares Regression — the payoff of the projection idea.