Pseudoinverse: Exercises — Maths deep dive

Before we start, three reminders you will lean on constantly:

Definition Notation and terms you will use throughout

$I_{k}$ means the $k \times k$ identity matrix — 1's on the diagonal, 0's elsewhere; it leaves any vector unchanged. When the size is clear I sometimes just write $I$ .
Independent columns/rows means none of them can be written as a combination of the others — none is redundant.
The rank $r$ of $A$ is the number of independent columns (equivalently, independent rows), and equals the number of nonzero singular values. "Full column rank" means $r = n$ ; "full row rank" means $r = m$ .
The nullspace of $A$ is the set of all vectors $v$ that $A$ crushes to zero, i.e. $A v = 0$ . These are the "invisible" directions: adding any nullspace vector to a solution of $A x = b$ changes $x$ but not $A x = b$ . The nullspace is exactly the wasted-energy directions the minimum-norm solution avoids.
The gradient $\nabla E$ of a scalar function $E (x)$ is the vector of its slopes in each coordinate direction — it points "uphill", and it is the zero vector exactly at a flat spot (a minimum, for a bowl-shaped function).
The four Penrose conditions that uniquely pin down $A^{+}$ are, in order:
1. $A A^{+} A = A$
2. $A^{+} A A^{+} = A^{+}$
3. $(A A^{+})^{⊤} = A A^{+}$ (so $A A^{+}$ is symmetric)
4. $(A^{+} A)^{⊤} = A^{+} A$ (so $A^{+} A$ is symmetric)
When I say "Condition 1/2/3/4" below, I mean exactly these.

Level 1 — Recognition

L1.1 — Which formula?

For each matrix, state its shape, whether it has full column rank, full row rank, or neither, and which of the three formulas you would reach for.

$A = 103210$
$B = (100121)$
$C = (1224)$

Recall Solution

$A$ is $3 \times 2$ (tall). Its two columns $(1, 0, 3)$ and $(2, 1, 0)$ are not multiples of each other, so they are independent → full column rank ( $r = 2 = n$ ). Use $A^{+} = (A^{⊤} A)^{- 1} A^{⊤}$ .

$B$ is $2 \times 3$ (fat). Its two rows $(1, 0, 2)$ and $(0, 1, 1)$ are independent → full row rank ( $r = 2 = m$ ). Use $B^{+} = B^{⊤} (B B^{⊤})^{- 1}$ .

$C$ is $2 \times 2$ (square) but row 2 $= 2 \times$ row 1, so rank $r = 1$ , not full. Neither $C^{⊤} C$ nor $C C^{⊤}$ is invertible → must use the SVD formula $C^{+} = V Σ^{+} U^{⊤}$ .

L1.2 — True or false

If $A$ is square and invertible then $A^{+} = A^{- 1}$ .
$A^{+} A$ always equals the identity.
The matrix $A A^{+}$ is symmetric.

Recall Solution

True — the pseudoinverse generalises the inverse; for invertible square $A$ they coincide.
False — $A^{+} A = I_{n}$ only for full column rank. In general $A^{+} A$ is an orthogonal projection onto the row space, and a projection can have eigenvalue $0$ .
True — this is Penrose Condition 3: $(A A^{+})^{⊤} = A A^{+}$ .

Level 2 — Application

L2.1 — Compute a tall pseudoinverse

Find $A^{+}$ for $A = 111012$ , then use it to solve the least-squares problem $A x = b$ with $b = 600$ .

Recall Solution

Shape check. $3 \times 2$ , columns independent → Case 1, $A^{+} = (A^{⊤} A)^{- 1} A^{⊤}$ .

Step 1 — $A^{⊤} A$ . ( $A^{⊤}$ has $A$ 's columns as its rows.) $A^{⊤} A = (101112) 111012 = (3335) .$ Step 2 — invert it. Determinant $= 3 \cdot 5 - 3 \cdot 3 = 6$ . For a $2 \times 2$ matrix $(a c b d)$ the inverse is $\frac{1}{a d - b c} (d - c - b a)$ : $(A^{⊤} A)^{- 1} = \frac{1}{6} (5 - 3 - 3 3) .$ Step 3 — multiply by $A^{⊤}$ . $A^{+} = \frac{1}{6} (5 - 3 - 3 3) (101112) = \frac{1}{6} (5 - 3 20 - 1 3) .$ Step 4 — solve. $x = A^{+} b = \frac{1}{6} (5 - 3 20 - 1 3) 600 = \frac{1}{6} (30 - 18) = (5 - 3) .$ So the best-fit line through $(0, 6), (1, 0), (2, 0)$ is $y = 5 - 3 t$ . No line hits all three points, so this is the least-squares answer — the one making $∥ A x - b ∥^{2}$ smallest.

L2.2 — Compute a fat pseudoinverse

Find $B^{+}$ for $B = (111)$ and the minimum-norm solution of $B x = (3)$ .

Recall Solution

Shape check. $1 \times 3$ (fat), one nonzero row → full row rank → Case 2, $B^{+} = B^{⊤} (B B^{⊤})^{- 1}$ .

Step 1 — $B B^{⊤}$ (a $1 \times 1$ number): $1 \cdot 1 + 1 \cdot 1 + 1 \cdot 1 = 3$ . Step 2 — invert: $(B B^{⊤})^{- 1} = \frac{1}{3}$ . Step 3 — assemble: $B^{+} = 111 \cdot \frac{1}{3} = 1/3 1/3 1/3 .$ Step 4 — solve: $x = B^{+} (3) = 111 .$ Why this is the minimum-norm one. Every solution of $x_{1} + x_{2} + x_{3} = 3$ can be written as $(1, 1, 1)$ plus a vector in the nullspace of $B$ — the directions $B$ crushes to zero, i.e. all $v$ with $v_{1} + v_{2} + v_{3} = 0$ (for example $(1, - 1, 0)$ ). Adding such a $v$ leaves $B x$ unchanged but only lengthens $x$ , because $(1, 1, 1)$ is perpendicular to every nullspace vector: $(1, 1, 1) \cdot (1, - 1, 0) = 0$ . So $(1, 1, 1)$ is the shortest — the one with no wasted nullspace component.

Level 3 — Analysis

L3.1 — Rank-deficient by SVD

Compute $A^{+}$ for $A = (3000)$ using the SVD formula, and verify $A A^{+} A = A$ .

Recall Solution

Read off the SVD. $A$ is already diagonal, so $U = V = I_{2}$ (the $2 \times 2$ identity), $Σ = (3000)$ . The singular values are $σ_{1} = 3$ , $σ_{2} = 0$ (so rank $r = 1$ ).

Build $Σ^{+}$ . Flip nonzero, leave zero: $Σ^{+} = (1/3 0 00)$ . $A^{+} = V Σ^{+} U^{⊤} = (1/3 0 00) .$ Verify Condition 1 ( $A A^{+} A = A$ ). $A A^{+} = (1000)$ (projection onto the first axis), so $A A^{+} A = (1000) (3000) = (3000) = A$ ✓. The zero direction stays dead: no information there to invert.

L3.2 — Projection viewpoint

For the tall $A$ of L2.1, compute $P = A A^{+}$ and confirm it is (a) symmetric and (b) idempotent ( $P^{2} = P$ ). What does $P$ do geometrically?

Recall Solution

$A^{+} = \frac{1}{6} (5 - 3 20 - 1 3)$ . Then $P = A A^{+} = 111012 \frac{1}{6} (5 - 3 20 - 1 3) = \frac{1}{6} 52 - 1 222 - 1 25 .$ (a) Symmetric: the matrix equals its transpose (check the off-diagonals: $2 = 2$ , $- 1 = - 1$ ) ✓ — this is Penrose Condition 3. (b) Idempotent — proved, not just checked. "Idempotent" means applying $P$ twice does the same as once: $P^{2} = P$ . Here is why it must hold for any $A A^{+}$ built from a genuine pseudoinverse, using only the Penrose conditions: $P^{2} = (A A^{+}) (A A^{+}) = A Condition 2 (A^{+} A A^{+}) = A A^{+} = P .$ The middle grouping used Condition 2 ( $A^{+} A A^{+} = A^{+}$ ). So every orthogonal projection of this form satisfies $P^{2} = P$ — projecting an already-projected vector changes nothing. (A direct numeric check for this specific $P$ is also in VERIFY.) Geometry. $P$ is the orthogonal projection onto the column space of $A$ — a 2D plane inside 3D. The figure below shows exactly this: the blue plane is the column space (all vectors $A x$ ), the yellow arrow is $b = (6, 0, 0)$ , and $P$ drops $b$ straight down onto the plane to the green arrow $P b = A x = (5, 2, - 1)$ . The red segment is the leftover residual $b - P b$ , and it meets the plane at a right angle. That right angle is the least-squares condition $A^{⊤} (b - A x) = 0$ : the error is smallest exactly when it is perpendicular to the plane.

Figure L3.2 — Least-squares as projection. Blue plane: the column space of $A$ (every possible $A x$ ). Yellow arrow $b$ : the target we cannot reach. Green arrow $P b = A x = (5, 2, - 1)$ : the closest point in the plane, produced by $A A^{+}$ . Red segment: the residual $b - P b$ , perpendicular to the plane — this perpendicularity is what "least squares" means.

Level 4 — Synthesis

L4.1 — Build $A^{+}$ two ways and compare

For $A = 200030$ , compute $A^{+}$ by (i) the tall formula and (ii) the SVD formula. Show they agree.

Recall Solution

(i) Tall formula. $A^{⊤} A = (4009)$ , inverse $(1/4 0 0 1/9)$ . $A^{+} = (A^{⊤} A)^{- 1} A^{⊤} = (1/4 0 0 1/9) (200300) = (1/2 0 0 1/3 00) .$ (ii) SVD. $A$ 's nonzero entries are already the singular values along the diagonal: $σ_{1} = 2, σ_{2} = 3$ , with $U = I_{3}, V = I_{2}$ . So $Σ = A$ , and $Σ^{+} = (1/2 0 0 1/3 00) = A^{+} .$ Both routes give the same matrix ✓. When full column rank and an easy SVD both apply, they must agree — the pseudoinverse is unique.

L4.2 — A full non-diagonal SVD pseudoinverse

Compute $A^{+}$ for the rank-1 matrix $A = (1111)$ .

Recall Solution

Step 1 — form $A^{⊤} A$ and find its eigenvalues. $A^{⊤} A = (2222)$ . Its eigenvalues solve $det (A^{⊤} A - λ I_{2}) = (2 - λ)^{2} - 4 = 0$ , giving $λ = 4$ or $λ = 0$ . Singular values are square roots: $σ_{1} = 4 = 2$ , $σ_{2} = 0 = 0$ (so rank $r = 1$ ).

Step 2 — right singular vectors $v_{i}$ (eigenvectors of $A^{⊤} A$ , unit length). For $λ = 4$ : solve $(2 - 4 2 2 2 - 4) v = 0 \Rightarrow - 2 v_{1} + 2 v_{2} = 0 \Rightarrow$ direction $(1, 1)$ . Normalise (divide by its length $2$ ): $v_{1} = \frac{1}{2} (1, 1)$ . For $λ = 0$ : direction $(1, - 1)$ , normalise: $v_{2} = \frac{1}{2} (1, - 1)$ . Check orthonormal: $v_{1} \cdot v_{1} = \frac{1}{2} (1 + 1) = 1$ , $v_{2} \cdot v_{2} = 1$ , and $v_{1} \cdot v_{2} = \frac{1}{2} (1 - 1) = 0$ ✓ — so $V = [v_{1} v_{2}]$ is orthogonal.

Step 3 — left singular vector $u_{1}$ (only the nonzero $σ$ has one). Rule: $u_{i} = \frac{1}{σ _{i}} A v_{i}$ . $u_{1} = \frac{1}{2} A v_{1} = \frac{1}{2} \cdot \frac{1}{2} (1111) (11) = \frac{1}{2} \cdot \frac{1}{2} (22) = \frac{1}{2} (1, 1) .$ Check unit length: $u_{1} \cdot u_{1} = \frac{1}{2} (1 + 1) = 1$ ✓.

Step 4 — assemble $A^{+}$ . The SVD sum runs only over nonzero singular values, so with $r = 1$ : $A^{+} = \frac{1}{σ _{1}} v_{1} u_{1}^{⊤} = \frac{1}{2} \cdot \frac{1}{2} (11) \cdot \frac{1}{2} (11) = \frac{1}{2} \cdot \frac{1}{2} (1111) = (1/4 1/4 1/4 1/4) .$ Sanity via Condition 1: $A A^{+} = (1/2 1/2 1/2 1/2)$ (projection onto the line $(1, 1)$ ), and $A A^{+} A = A$ ✓.

Level 5 — Mastery

L5.1 — Prove the Penrose conditions from the tall formula

Let $A$ have full column rank and $A^{+} = (A^{⊤} A)^{- 1} A^{⊤}$ . Prove all four Penrose conditions hold.

Recall Solution

Write $G = (A^{⊤} A)^{- 1}$ . First note $G$ is symmetric: $A^{⊤} A$ is symmetric (since $(A^{⊤} A)^{⊤} = A^{⊤} A$ ), and the inverse of a symmetric matrix is symmetric because $(M^{- 1})^{⊤} = (M^{⊤})^{- 1} = M^{- 1}$ when $M^{⊤} = M$ . Hence $G^{⊤} = G$ .

Condition 4 first: $A^{+} A = (A^{⊤} A)^{- 1} A^{⊤} A = G (A^{⊤} A) = I_{n}$ . The identity is symmetric ✓ — and this also shows $A^{+}$ is a genuine left inverse.

Condition 1: $A A^{+} A = A (A^{+} A) = A I_{n} = A$ ✓.

Condition 2: $A^{+} A A^{+} = (A^{+} A) A^{+} = I_{n} A^{+} = A^{+}$ ✓.

Condition 3: $A A^{+} = A G A^{⊤}$ . Transpose it: $(A G A^{⊤})^{⊤} = A G^{⊤} A^{⊤} = A G A^{⊤}$ (using $G^{⊤} = G$ ), so $A A^{+}$ is symmetric ✓.

All four hold, and since the Penrose conditions define a unique matrix, this really is the pseudoinverse.

L5.2 — Least squares from scratch, then read off $A^{+}$

Derive $x = (A^{⊤} A)^{- 1} A^{⊤} b$ by minimising $∥ A x - b ∥^{2}$ , explaining why the gradient is set to zero, and hence identify $A^{+}$ . Then verify numerically for the L2.1 data that $x = (5, - 3)$ has smaller error than the nearby $x = (4, - 2)$ .

Recall Solution

Set up the error. $∥ A x - b ∥^{2}$ means "squared length of the residual vector $A x - b$ ". Expand using $∥ v ∥^{2} = v^{⊤} v$ : $E (x) = (A x - b)^{⊤} (A x - b) = x^{⊤} A^{⊤} A x - 2 x^{⊤} A^{⊤} b + b^{⊤} b .$ Why set the gradient to zero? Recall from the definitions box: the gradient $\nabla E$ is the vector of slopes of $E$ in each coordinate direction. Here $E$ is a smooth, bowl-shaped (convex) function of $x$ — like a parabola in many dimensions. Its lowest point is exactly where the slope in every direction is flat, i.e. where the gradient is the zero vector. This is the least-squares condition. $\nabla E = 2 A^{⊤} A x - 2 A^{⊤} b = 0 \Rightarrow A^{⊤} A x = A^{⊤} b .$ These are the normal equations. With full column rank $A^{⊤} A$ is invertible: $x = (A^{⊤} A)^{- 1} A^{⊤} b \Rightarrow A^{+} = (A^{⊤} A)^{- 1} A^{⊤} .$ Geometrically: the residual $b - A x$ is orthogonal to every column of $A$ , i.e. $A^{⊤} (b - A x) = 0$ — the same equation. That is why $A A^{+}$ is the orthogonal projection onto the column space.

Numeric check (L2.1 data, $A = 111012$ , $b = (6, 0, 0)$ ).

At the pseudoinverse answer $x = (5, - 3)$ : $A x = 1 \cdot 5 + 0 \cdot (- 3) 1 \cdot 5 + 1 \cdot (- 3) 1 \cdot 5 + 2 \cdot (- 3) = (5, 2, - 1)$ , so residual $b - A x = (6 - 5, 0 - 2, 0 - (- 1)) = (1, - 2, 1)$ , and $∥ b - A x ∥^{2} = 1^{2} + (- 2)^{2} + 1^{2} = 1 + 4 + 1 = 6$ .
At the nearby $x = (4, - 2)$ : $A x = (4, 2, 0)$ , residual $(6 - 4, 0 - 2, 0 - 0) = (2, - 2, 0)$ , and $∥ b - A x ∥^{2} = 2^{2} + (- 2)^{2} + 0^{2} = 4 + 4 + 0 = 8$ . Since $6 < 8$ , the pseudoinverse solution $(5, - 3)$ genuinely has the smaller squared error — no other choice can beat it, because it is the unique minimum of the convex bowl $E (x)$ ✓.

Recall One-line self-test

Given any matrix, my first move is to check ::: its rank and shape, then pick tall / fat / SVD. $A A^{+}$ geometrically is ::: orthogonal projection onto the column space of $A$ . The SVD sum for $A^{+}$ has how many terms? ::: exactly $r$ , the rank of $A$ (one term per nonzero singular value). The nullspace of $A$ is ::: the set of all $v$ with $A v = 0$ — the directions $A$ crushes to zero.

Connections

Pseudoinverse — the parent note these exercises drill.
Singular Value Decomposition — the engine behind L3, L4, L5.
Least Squares Regression — the story behind L2.1 and L5.2.
Orthogonal Projection — the meaning of $A A^{+}$ in L3.2.
Four Fundamental Subspaces — why nullspace directions get killed.
Matrix Inverse — the special case $A^{+} = A^{- 1}$ .
Eigenvalues and Eigenvectors — $σ_{i}^{2}$ are eigenvalues of $A^{⊤} A$ (used in L4.2).