A normal inverse A − 1 A^{-1} A − 1 A A A square and invertible . But most real matrices are rectangular (m × n m \times n m × n A A A A + A^+ A + best possible substitute for an inverse: it gives the least-squares, minimum-norm answer to A x = b Ax = b A x = b
WHY it matters: every overdetermined system (more equations than unknowns — data fitting) and every underdetermined system (more unknowns than equations — many solutions) is solved by A + A^+ A +
Definition Moore–Penrose pseudoinverseFor any real matrix A ∈ R m × n A \in \mathbb{R}^{m\times n} A ∈ R m × n A + ∈ R n × m A^+ \in \mathbb{R}^{n\times m} A + ∈ R n × m unique matrix satisfying the four Penrose conditions :
A A + A = A AA^+A = A A A + A = A A + A A + = A + A^+AA^+ = A^+ A + A A + = A + ( A A + ) ⊤ = A A + (AA^+)^\top = AA^+ ( A A + ) ⊤ = A A + A A + AA^+ A A + ( A + A ) ⊤ = A + A (A^+A)^\top = A^+A ( A + A ) ⊤ = A + A A + A A^+A A + A
When A A A A + = A − 1 A^+ = A^{-1} A + = A − 1 generalizes the inverse.
The key geometric fact: A A + AA^+ A A + orthogonal projection onto the column space of A A A A + A A^+A A + A row space of A A A
We want to solve A x = b Ax = b A x = b b b b no exact solution exists — so we minimize the error ∥ A x − b ∥ 2 \|Ax - b\|^2 ∥ A x − b ∥ 2
Now A x = b Ax=b A x = b infinitely many solutions. We pick the one with smallest norm ∥ x ∥ \|x\| ∥ x ∥ — Why? It's the unique "no wasted energy" solution, orthogonal to the nullspace.
Worked example Example 1 — tall matrix, least squaresA = ( 1 0 1 1 1 2 ) A=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix} A = 1 1 1 0 1 2 b = ( 6 0 0 ) b=\begin{pmatrix}6\\0\\0\end{pmatrix} b = 6 0 0 y = c + d t y = c + d t y = c + d t t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2
Step 1: A ⊤ A = ( 3 3 3 5 ) A^\top A=\begin{pmatrix}3&3\\3&5\end{pmatrix} A ⊤ A = ( 3 3 3 5 ) Why? Full column rank ⇒ use Case 1.
Step 2: A ⊤ b = ( 6 0 ) A^\top b = \begin{pmatrix}6\\0\end{pmatrix} A ⊤ b = ( 6 0 ) Step 3: Solve ( 3 3 3 5 ) x = ( 6 0 ) \begin{pmatrix}3&3\\3&5\end{pmatrix}x=\begin{pmatrix}6\\0\end{pmatrix} ( 3 3 3 5 ) x = ( 6 0 ) = 6 =6 = 6 x = 1 6 ( 5 − 3 − 3 3 ) ( 6 0 ) = ( 5 − 3 ) x=\frac{1}{6}\begin{pmatrix}5&-3\\-3&3\end{pmatrix}\begin{pmatrix}6\\0\end{pmatrix}=\begin{pmatrix}5\\-3\end{pmatrix} x = 6 1 ( 5 − 3 − 3 3 ) ( 6 0 ) = ( 5 − 3 ) y = 5 − 3 t y = 5 - 3t y = 5 − 3 t Why this answer and not exact? No line passes through ( 0 , 6 ) , ( 1 , 0 ) , ( 2 , 0 ) (0,6),(1,0),(2,0) ( 0 , 6 ) , ( 1 , 0 ) , ( 2 , 0 ) A + A^+ A +
Worked example Example 2 — fat matrix, minimum normA = ( 1 1 ) A = \begin{pmatrix}1 & 1\end{pmatrix} A = ( 1 1 ) b = ( 2 ) b = (2) b = ( 2 ) x 1 + x 2 = 2 x_1+x_2=2 x 1 + x 2 = 2 ∥ x ∥ \|x\| ∥ x ∥
Step 1: Full row rank ⇒ Case 2. A A ⊤ = 2 AA^\top = 2 A A ⊤ = 2 Step 2: A + = A ⊤ ( A A ⊤ ) − 1 = ( 1 1 ) ⋅ 1 2 = ( 1 / 2 1 / 2 ) A^+ = A^\top(AA^\top)^{-1} = \begin{pmatrix}1\\1\end{pmatrix}\cdot\frac12 = \begin{pmatrix}1/2\\1/2\end{pmatrix} A + = A ⊤ ( A A ⊤ ) − 1 = ( 1 1 ) ⋅ 2 1 = ( 1/2 1/2 ) Step 3: x = A + b = ( 1 1 ) x = A^+b = \begin{pmatrix}1\\1\end{pmatrix} x = A + b = ( 1 1 ) Why? Of all ( x 1 , x 2 ) (x_1,x_2) ( x 1 , x 2 ) ( 1 , 1 ) (1,1) ( 1 , 1 ) ( 1 , − 1 ) (1,-1) ( 1 , − 1 )
Worked example Example 3 — rank-deficient via SVDA = ( 2 0 0 0 ) A=\begin{pmatrix}2&0\\0&0\end{pmatrix} A = ( 2 0 0 0 ) σ 1 = 2 , σ 2 = 0 \sigma_1=2,\sigma_2=0 σ 1 = 2 , σ 2 = 0 Σ + = ( 1 / 2 0 0 0 ) \Sigma^+ = \begin{pmatrix}1/2 & 0\\0 & 0\end{pmatrix} Σ + = ( 1/2 0 0 0 ) U = V = I U=V=I U = V = I A + = ( 1 / 2 0 0 0 ) A^+ = \begin{pmatrix}1/2&0\\0&0\end{pmatrix} A + = ( 1/2 0 0 0 ) Why the zero stays? The second direction is in the nullspace — no information to invert.
Common mistake Steel-manned common errorsMistake 1: "A + = ( A ⊤ A ) − 1 A ⊤ A^+ = (A^\top A)^{-1}A^\top A + = ( A ⊤ A ) − 1 A ⊤
Why it feels right: it's the formula in every least-squares textbook. The catch: it needs full column rank , else A ⊤ A A^\top A A ⊤ A Fix: use the SVD formula for rank-deficient matrices.
Mistake 2: "A + A = I A^+A = I A + A = I
Why it feels right: an inverse undoes things. Truth: A + A = I A^+A=I A + A = I A + A A^+A A + A projection , not identity (it can have eigenvalue 0).
Mistake 3: "Invert the zero singular values too (use a tiny number)."
Why it feels tempting: avoid the awkward zero. Danger: 1 / ϵ 1/\epsilon 1/ ϵ Fix: truly zero (or negligible) σ i \sigma_i σ i 0 0 0
Recall Feynman: explain to a 12-year-oldImagine a vending machine. A normal inverse is a machine that, given a snack, perfectly hands back the exact coins you put in. But some machines are broken — they squish information so you can't perfectly get your coins back, or there are many coin-combos that give the same snack. The pseudoinverse is the fairest possible refund machine : if a perfect refund is impossible, it gives the answer that's closest (least error); if many refunds work, it gives the smallest, simplest one (least wasted coins). It never crashes, no matter the machine.
Mnemonic Remember the three formulas"Tall Tops, Fat Bottoms, SVD always."
Tall (full column rank): ( A ⊤ A ) − 1 (A^\top A)^{-1} ( A ⊤ A ) − 1 top-left → A + = ( A ⊤ A ) − 1 A ⊤ A^+=(A^\top A)^{-1}A^\top A + = ( A ⊤ A ) − 1 A ⊤ Fat (full row rank): A ⊤ A^\top A ⊤ bottom → A + = A ⊤ ( A A ⊤ ) − 1 A^+=A^\top(AA^\top)^{-1} A + = A ⊤ ( A A ⊤ ) − 1 SVD (V Σ + U ⊤ V\Sigma^+U^\top V Σ + U ⊤ any case — flip U , V U,V U , V σ \sigma σ
What are the four Penrose conditions? A A + A = A AA^+A=A A A + A = A ;
A + A A + = A + A^+AA^+=A^+ A + A A + = A + ;
( A A + ) ⊤ = A A + (AA^+)^\top=AA^+ ( A A + ) ⊤ = A A + ;
( A + A ) ⊤ = A + A (A^+A)^\top=A^+A ( A + A ) ⊤ = A + A .
Pseudoinverse for full column rank? A + = ( A ⊤ A ) − 1 A ⊤ A^+=(A^\top A)^{-1}A^\top A + = ( A ⊤ A ) − 1 A ⊤ (a left inverse,
A + A = I A^+A=I A + A = I ).
Pseudoinverse for full row rank? A + = A ⊤ ( A A ⊤ ) − 1 A^+=A^\top(AA^\top)^{-1} A + = A ⊤ ( A A ⊤ ) − 1 (a right inverse,
A A + = I AA^+=I A A + = I ).
Universal SVD formula for A + A^+ A + A + = V Σ + U ⊤ A^+ = V\Sigma^+U^\top A + = V Σ + U ⊤ , inverting each nonzero
σ i \sigma_i σ i , leaving zeros as zeros.
What does A A + AA^+ A A + Orthogonal projection onto the column space of
A A A .
Why does least squares give A ⊤ A x = A ⊤ b A^\top Ax=A^\top b A ⊤ A x = A ⊤ b Setting
∇ ∥ A x − b ∥ 2 = 0 \nabla\|Ax-b\|^2=0 ∇∥ A x − b ∥ 2 = 0 ; the residual is orthogonal to the column space.
When does A + = A − 1 A^+=A^{-1} A + = A − 1 When
A A A is square and invertible.
What problem does the fat-matrix pseudoinverse solve? The minimum-norm solution among infinitely many solutions of
A x = b Ax=b A x = b .
Why not invert zero singular values? 1 / 0 1/0 1/0 is undefined and tiny values blow up noise; those directions carry no information.
Inverse A^-1 needs square invertible
Intuition Hinglish mein samjhoDekho, normal inverse A − 1 A^{-1} A − 1 pseudoinverse A + A^+ A + kehte hain. Yeh hamesha exist karta hai, kabhi crash nahi karta.
Do main situations samjho. Agar matrix tall hai (full column rank), toh exact solution nahi milta, isliye hum error ∥ A x − b ∥ 2 \|Ax-b\|^2 ∥ A x − b ∥ 2 least squares , aur formula banta hai A + = ( A ⊤ A ) − 1 A ⊤ A^+=(A^\top A)^{-1}A^\top A + = ( A ⊤ A ) − 1 A ⊤ fat hai (full row rank), toh infinite solutions hote hain, aur hum chhote se chhota ∥ x ∥ \|x\| ∥ x ∥ A + = A ⊤ ( A A ⊤ ) − 1 A^+=A^\top(AA^\top)^{-1} A + = A ⊤ ( A A ⊤ ) − 1
Sabse powerful tareeka hai SVD : A = U Σ V ⊤ A=U\Sigma V^\top A = U Σ V ⊤ A + = V Σ + U ⊤ A^+ = V\Sigma^+U^\top A + = V Σ + U ⊤ σ i \sigma_i σ i 1 / σ i 1/\sigma_i 1/ σ i
Geometrically samjho: A A + AA^+ A A + projection hai column space par, aur A + A A^+A A + A
