Pseudoinverse: Question bank — Maths deep dive

Before we start, here is every symbol and fact this page leans on. Read it once; the traps below assume all of it.

Recall Quick symbol refresher (shapes, ranks, subspaces)

$A \in R^{m \times n}$ means $A$ has $m$ rows, $n$ columns. Tall = more rows than columns ( $m > n$ ); fat = more columns than rows ( $m < n$ ).
$A^{⊤}$ is the transpose (flip across the diagonal). $A^{- 1}$ is the ordinary inverse (only for square, non-singular $A$ ).
Rank = number of independent columns = number of independent rows. Full column rank means all $n$ columns are independent; full row rank means all $m$ rows are independent.
$∥ x ∥$ is the length of a vector; $∥ A x - b ∥^{2}$ is the squared error we minimise in least squares.
The four subspaces (see Four Fundamental Subspaces): column space (all outputs $A x$ ), row space, nullspace (inputs $A$ sends to $0$ ), left nullspace.

The two pictures below are the geometry every trap on this page secretly tests: look at them before answering.

True or false — justify

A^{+}

exists for every real matrix, including the zero matrix.

True. The Singular Value Decomposition gives

A = U Σ V^{⊤}

for any shape and rank, so

A^{+} = V Σ^{+} U^{⊤}

always exists; the zero matrix's pseudoinverse is again the zero matrix (nothing to invert).

A

is square and invertible, then

A^{+} = A^{- 1}

True. With full rank and square shape,

A^{⊤} A

is invertible and

(A^{⊤} A)^{- 1} A^{⊤} = A^{- 1} (A^{⊤})^{- 1} A^{⊤} = A^{- 1}

; the Matrix Inverse is a special case of the pseudoinverse.

A^{+} A = I

for every matrix

A

False.

A^{+} A = I

only when

A

has full column rank. In general

A^{+} A

is the orthogonal projection onto the row space, which can have eigenvalue

0

along nullspace directions.

A A^{+}

and

A^{+} A

are always symmetric.

True. Penrose conditions 3 and 4 demand exactly

(A A^{+})^{⊤} = A A^{+}

and

(A^{+} A)^{⊤} = A^{+} A

; symmetry is what makes them orthogonal projections rather than skewed ones.

(A^{+})^{+} = A

for every matrix.

True. Taking the pseudoinverse twice returns the original — the four Penrose conditions are symmetric in the roles of

A

and

A^{+}

, so

A

satisfies them for

A^{+}

(full "why" in the Edge-cases section).

A^{+}

is always the same shape as

A

False. If

A

m \times n

, then

A^{+}

n \times m

— the shape is transposed so that the products

A A^{+}

(

m \times m

) and

A^{+} A

(

n \times n

) make sense.

For a fat, full-row-rank matrix,

A A^{+} = I_{m}

but

A^{+} A \neq = I_{n}

True.

A^{+}

is a right inverse here:

A A^{+} = I_{m}

. But

A^{+} A

is only a projection onto the row space (dimension

m < n

), so it cannot be the full

n \times n

identity.

The formula

A^{+} = (A^{⊤} A)^{- 1} A^{⊤}

works for any tall matrix.

False. It needs full column rank. A tall matrix can still have dependent columns, making

A^{⊤} A

singular and non-invertible — then you must use the SVD formula.

If every singular value of

A

is nonzero,

A^{+} A = I

True but read carefully. All nonzero singular values means full column rank (rank

= n

), which gives

A^{+} A = I_{n}

. It does not force

A A^{+} = I

unless

A

is also full row rank (i.e. square).

Spot the error

A

3 \times 2

with rank 1, so I'll use

A^{+} = (A^{⊤} A)^{- 1} A^{⊤}

Error: rank 1 for a 2-column matrix means the columns are dependent, so

A^{⊤} A

is singular — you cannot invert it. Use

A^{+} = V Σ^{+} U^{⊤}

from the Singular Value Decomposition instead.

"A singular value is

1 0^{- 9}

, basically zero, so I'll invert it to

1 0^{9}

to be safe."

Error: inverting a tiny

σ

produces a huge

1/ σ

that massively amplifies noise in that direction. Treat negligible singular values as exactly

0

so they map to

0

(see Mistake 3 in the parent note).

"Since

A A^{+} A = A

, cancelling the

A

on the right gives

A^{+} A = I

Error: "cancelling" means multiplying both sides by an inverse to remove a factor — but

A

has no inverse when it is not square-and-invertible, so there is nothing to multiply by.

A A^{+} A = A

is a projection identity (

A A^{+}

projects onto the column space); it does not imply

A^{+} A = I

unless

A

has full column rank.

A^{+}

is the inverse, so

A A^{+} b = b

for any

b

Error:

A A^{+}

is the orthogonal projection onto the column space, so

A A^{+} b = b

only when

b

already lies in the column space. Otherwise it returns the closest point in the column space, which is exactly why least squares needs

A^{+}

"To find

Σ^{+}

I invert every diagonal entry of

Σ

Error: you invert only the nonzero singular values; zeros stay zero. You must also transpose the shape (an

m \times n

block becomes

n \times m

) so the product

V Σ^{+} U^{⊤}

has matching dimensions.

"For

A x = b

with infinitely many solutions,

A^{+} b

gives some random valid solution."

Error:

A^{+} b

gives the specific solution of minimum norm

∥ x ∥

— the one lying entirely in the row space, orthogonal to the nullspace. It is uniquely determined, not arbitrary.

σ_{i}

are the eigenvalues of

A^{⊤} A

Error: the squares

σ_{i}^{2}

are the eigenvalues of

A^{⊤} A

(see Eigenvalues and Eigenvectors). The singular values themselves are their non-negative square roots.

Why questions

Why do we minimise

∥ A x - b ∥^{2}

rather than

∥ A x - b ∥

in least squares?

The squared length is a smooth convex function of

x

, so its gradient is simple (

2 A^{⊤} A x - 2 A^{⊤} b

) and vanishes at exactly one minimum; minimising the un-squared norm gives the identical answer but with a messier, non-differentiable-at-zero expression.

Why does setting the gradient to zero give

A^{⊤} A x = A^{⊤} b

At the minimum the residual

A x - b

must be orthogonal to the column space, i.e.

A^{⊤} (A x - b) = 0

. This is the geometric heart of Least Squares Regression: the error points straight out of the space of achievable outputs.

Why can we write

x = A^{⊤} w

for the minimum-norm solution of a fat system?

Forcing

x

into the row space (the range of

A^{⊤}

) removes any component in the nullspace. Nullspace components add length without changing

A x

, so discarding them is exactly what makes

∥ x ∥

smallest.

Why is the SVD called the "universal" pseudoinverse formula?

It never asks for

A^{⊤} A

A A^{⊤}

to be invertible — it works for any shape and any rank by inverting only the nonzero singular values, so it covers full-column-rank, full-row-rank, and rank-deficient cases in one stroke.

Why is

A A^{+}

a projection and not the identity in general?

A A^{+}

can only reach outputs inside the column space; if that space is smaller than all of

R^{m}

, some directions get flattened to their nearest in-space point, and any map that flattens directions has eigenvalue

0

, so it cannot be

I

Why do the two special-case formulas

(A^{⊤} A)^{- 1} A^{⊤}

and

A^{⊤} (A A^{⊤})^{- 1}

agree with the SVD formula?

Substituting

A = U Σ V^{⊤}

into either formula and simplifying (using

U^{⊤} U = I

V^{⊤} V = I

) collapses it to

V Σ^{+} U^{⊤}

; they are the same object written for the case where the relevant Gram matrix happens to be invertible.

Why must the residual be orthogonal to the column space at the least-squares optimum?

If the residual had any component inside the column space, we could move

x

to cancel that component and shrink the error — so at the true minimum no such component remains, leaving the residual perpendicular.

Edge cases

Sketch why

(A^{+})^{+} = A

Write out the four Penrose conditions for the pair

(A^{+}, A)

instead of

(A, A^{+})

: condition 1 becomes

A^{+} A A^{+} = A^{+}

, condition 2 becomes

A A^{+} A = A

, and conditions 3–4 swap into each other — all four still hold because they held for

(A, A^{+})

. Since

A

satisfies the four defining conditions of "the pseudoinverse of

A^{+}

", and that matrix is unique,

A

(A^{+})^{+}

What is

A^{+}

when

A

is the

n \times n

zero matrix?

The

n \times n

zero matrix. Every singular value is

0

, so

Σ^{+}

is all zeros and

A^{+} = V Σ^{+} U^{⊤} = 0

— there is no information to invert in any direction.

What is

0^{+}

for a rectangular zero matrix (

m \times n

The

n \times m

zero matrix. Shape flips (as always) but every entry is still

0

, since no direction carries invertible information.

For a rank-deficient square matrix, does

A^{+} = A^{- 1}

No.

A^{- 1}

does not exist for a singular matrix.

A^{+}

still exists via the SVD and acts as an inverse only on the row/column spaces, mapping the nullspace directions to

0

A

has a nullspace, what does

A^{+}

do to vectors in the left nullspace of

A

It sends them to

0

. Left-nullspace directions of

A

are outside the column space, so

A^{+}

(which only inverts column-space directions back into the row space) has nothing to invert there.

A tall matrix has some singular values equal to zero. Which formula do you use, and how many zeros can there be?

Use the SVD formula

A^{+} = V Σ^{+} U^{⊤}

. The number of zero singular values equals

n - rank (A)

, which is exactly the dimension of the nullspace; each zero marks one nullspace direction that gets mapped to

0

. Even one such zero makes

A^{⊤} A

singular, so the tall-matrix formula

(A^{⊤} A)^{- 1} A^{⊤}

breaks down.

A

is a nonzero column vector

v

(

n \times 1

), what is

A^{+}

The row vector

A^{+} = \frac{v ^{⊤}}{∥ v ∥ ^{2}}

. It has full column rank (rank 1), so

A^{+} = (v^{⊤} v)^{- 1} v^{⊤}

, and it acts as projection onto the line through

v

via

v v^{+}

Connections

Singular Value Decomposition — the universal engine behind every edge case above.
Least Squares Regression — where the orthogonality-of-residual "why" lives.
Four Fundamental Subspaces — nullspace and column-space traps come from here.
Orthogonal Projection — explains why $A A^{+}$ and $A^{+} A$ are projections, not identities.
Matrix Inverse — the square-invertible special case.
Eigenvalues and Eigenvectors — the $σ_{i}^{2}$ -as-eigenvalues trap.