Visual walkthrough — Pseudoinverse

4.5.42 · D2 · Maths › Linear Algebra (Full) › Pseudoinverse

---

Step 1 — Ek matrix space ke saath aslaan karta kya hai?

KYA. Hum input world (left) aur output world (right) draw karte hain. Machine $A$ arrow $x$ ko cross le jaati hai.

KYUN. Kuch bhi "invert" karne se pehle, humein forward motion kaisi dikhti hai yeh picture karna hoga. Ek inverse bas "is arrow ke saath waapis chalo" hai. Agar forward motion information kho de, toh backward motion perfect nahi ho sakti — aur wahi loss poori wajah hai ki $A^{+}$ exist karta hai.

PICTURE. Figure dekho. Input vector $x$ (lavender) squeeze aur rotate hokar $Ax$ (coral) ban jaata hai jo tilted column-space sheet par rehta hai. Output world ka har point us sheet par nahi hota — woh unreachable points exactly wahan hain jahan mushkil shuru hoti hai.

Symbols mein, har piece padhte hue:

$\underset{\text{the machine}}{\underbrace{A}}\underset{\text{input, }n\text{ numbers}}{\underbrace{x}}=\underset{\text{output, lands on column space}}{\underbrace{Ax}}$

---

Step 2 — Problem: $b$ sheet se door hai

KYA. Hum $b$ (coral dot) ko tilted sheet ke oopar floating rakhte hain, aur sheet par sabse kareeb point ko $\hat{b}$ mark karte hain.

YEH IDEA KYUN, AUR NAHI. "Closest" ka ek fair matlab hai jab exact impossible ho. Closest matlab hai sabse choti distance — aur ek point se flat sheet tak sabse choti distance hamesha perpendicular ke saath measure ki jaati hai (koi bhi tedha raasta zyada lamba hota hai). Wahi perpendicular har cheez ka beej hai.

PICTURE. $b$ se $\hat{b}$ tak ka dashed coral line sheet se right angle par milta hai (chhota square). Us dashed line ki length woh error hai jo hum avoid nahi kar sakte — residual.

$\underset{\text{error arrow ki squared length}}{\underbrace{\Vert Ax-b{\Vert }^2}}⟶\text{ise jitna ho sake chhota karo}$

Hum length ko square isliye karte hain kyunki squaring awkward square-root ko hata deta hai aur ek smooth bowl-shaped function deta hai jise minimise karna aasaan hai — minimum waise bhi same jagah baitta hai.

---

Step 3 — Perpendicular condition (yahi hai orthogonal projection)

KYA. Hum kehte hain "$Ax-b$, $A$ ke saare columns ke perpendicular hai" ek clean line mein.

DOT PRODUCT / TRANSPOSE KYUN. Har column ke against perpendicularity test karne ke liye, hum residual ko $A^{\top }$ se maarte hain ($A^{\top }$ ki har row $A$ ka ek column hai). Saare zeros milna matlab "koi bhi leftover sheet ke saath lean nahi karta." Yeh Step 2 ke geometric right angle ka exact algebraic twin hai — Orthogonal Projection dekho.

PICTURE. Residual arrow (coral) mint sheet se oopar khada hai; columns $a_1,a_2$ usme flat pade hain. Residual aur har column ke beech ka har angle $90^{\circ }$ hai.

$\underset{\text{rows = }A\text{ ke columns}}{\underbrace{A^{\top }}}\underset{\text{residual arrow}}{\underbrace{(Ax-b)}}=\underset{\text{sheet ke saath koi leftover nahi}}{\underbrace{0}}$

Bracket expand karne se normal equations milti hain — "normal" purane zamane ka word hai "perpendicular" ke liye:

$\underset{\text{chhoti }n\times n\text{ machine}}{\underbrace{A^{\top }A}}x=\underset{b\text{ ka sheet par shadow}}{\underbrace{A^{\top }b}}$

---

Step 4 — Tall case: solve karo, aur $A^{+}$ paida hota hai

KYA. Hum $A^{\top }A$ undo karke $x$ isolate karte hain. Jo bhi ab $b$ multiply karta hai wahi pseudoinverse hai — wahi literally iski definition hai yahan.

YEH ALLOWED KYUN HAI. Independent columns exactly woh condition hai ki $A^{\top }A$ ki koi zero direction nahi hai, isliye iska ordinary inverse exist karta hai. Agar columns dependent hote, toh $A^{\top }A$ ek direction ko zero mein flat kar deta aur invert karne se mana kar deta — woh Step 6 hai.

PICTURE. $A^{+}$ (mint arrow) floating $b$ se waapis input world mein jaata hai, least-squares answer $\hat{x}$ par land karta hai. Yeh ek genuine left inverse hai: $A$ phir $A^{+}$ karne se tum exactly waapis aate ho, kyunki

$A^{+}A=(A^{\top }A{)}^{-1}\underset{\text{cancel hota hai}}{\underbrace{A^{\top }A}}=I_n.$

Yeh precisely Least Squares Regression hai: best-fit line yahan rehti hai.

---

Step 5 — Fat case: bahut saare answers, sabse chota chuno

KYA. Saare valid $x$ mein se, hum woh single out karte hain jiska sabse chota length $\Vert x\Vert$ ho.

"SHORTEST" KYUN. Saare solutions sirf null space mein vectors se differ karte hain (woh directions jinhein $A$ zero mein crush karta hai — free, wasted energy; Four Fundamental Subspaces dekho). Saari null-space content hatane se woh unique piece bachti hai jo row space mein hoti hai. Trick $x=A^{\top }w$ automatically $x$ ko row space mein force karti hai, junk khatam karte hue. Shortest = solution line ke perpendicular.

PICTURE. Green solution line space mein drift karti hai; mint dot us par origin ke sabse kareeb point hai, line ki direction (null-space arrow) ke perpendicular baitha hua.

Yahan $A^{+}$ ek right inverse hai: $A{A}^{+}=A{A}^{\top }(A{A}^{\top }{)}^{-1}=I_m$.

---

Step 6 — Degenerate case: ek squashed direction

KYA. Hum machine ko poori ek axis ko zero mein flatten karte hue picture karte hain — ek pancake.

HUM FAKE KYU NAHI KAR SAKTE. Koi hamesha suggest karta hai "zero ki jagah bas ek chhote $ϵ$ se divide karo." Lekin $1/ϵ$ blast karta hai, noise ko sky-high uda deta hai (parent Mistake 3). Sachchi move: ek crushed direction mein koi information nahi hoti, isliye iska inverse contribution exactly zero hai.

PICTURE. Input square (lavender) ek single line (coral) par flat press ho jaata hai. Vertical direction hamesha ke liye chali gayi — koi honest arrow use waapis nahi la sakta.

---

Step 7 — SVD: universal machine, direction by direction

KYA. Kyunki $U,V$ pure rotations hain, unhe undo karna bas waapis rotate karna hai ($U^{\top }$, $V$). Sirf stretch step ko care ki zaroorat hai.

SIRF NONZEROS INVERT KYUN. Diagonal $\Sigma$ mein, ek nonzero stretch ${\sigma }_i$ ko $1/{\sigma }_i$ se shrink karke reverse kiya jaata hai. Ek zero stretch (Step 6 ki ek crushed direction) ka koi reverse nahi hota — use zero chhoddo. ${\Sigma }^{+}$ term by term padho:

${\sigma }_i\stackrel{\text{alive}}{\to }\frac{1}{{\sigma }_i},0\stackrel{\text{dead}}{\to }0.$

PICTURE. $A$ ke liye teen coloured stages left-to-right follow karo: rotate, stretch, rotate. Phir return path $A^{+}$ right-to-left chalta hai: unrotate, un-stretch (sirf live axes), unrotate. Dead axis dono taraf flat rehti hai.

Yeh single formula Step 4 aur Step 5 ko special cases ke roop mein reproduce karta hai, aur Step 6 se bachta bhi hai. Squared stretches ${\sigma }_i^2$, $A^{\top }A$ ke eigenvalues hain.

---

Ek-picture summary

Poori journey ek canvas par: $A$ ke through forward hum rotate, stretch, rotate karte hain aur ek direction kho sakte hain; $A^{+}$ ke through backward hum unrotate, live parts un-stretch karte hain, unrotate — sab kuch recover karte hain jo recoverable hai, aur kuch nahi.

Recall Feynman retelling — poori walkthrough plain words mein

Ek matrix ek machine hai jo tumhara arrow pakad kar reshape karti hai, use ek flat sheet par gira deti hai. Phir tum use apna arrow waapis maangne ko kehte ho.

Agar tumhara target sheet se door float karta hai (noisy data), toh perfect return impossible hai — isliye $A^{+}$ nearest sheet point tak seedha-neeche perpendicular giraa deta hai aur wahi return karta hai. Woh perpendicular least squares hai (Steps 2–4), tall matrices ke liye $A^{+}=(A^{\top }A{)}^{-1}{A}^{\top }$ deta hai.

Agar instead machine bahut saare correct returns allow kare (wide matrix), toh $A^{+}$ sabse chota, sabse aasaan wala choose karta hai — koi wasted null-space wandering nahi (Step 5), $A^{+}=A^{\top }(A{A}^{\top }{)}^{-1}$ deta hai.

Aur jab machine ek direction ko kuch bhi nahi karna mein flatten kare (Step 6), koi honest arrow use un-flatten nahi kar sakta — isliye $A^{+}$ use zero par chhodta hai. SVD (Step 7) yeh automatically karta hai: rotations undo karo, sirf living stretches invert karo, dead ones skip karo. Wahi hai $A^{+}=V{\Sigma }^{+}{U}^{\top }$ — ek formula sab par raaj karne ke liye.

Recall

Least-squares solution par residual kis ke perpendicular hota hai? ::: $A$ ke column space ke ($A$ ke har column ke). $A^{\top }A$ tall case mein invertible kyun hota hai lekin rank-deficient case mein nahi? ::: Independent columns koi zero direction nahi dete; rank deficiency ek aise crushed direction chhodti hai jise $A^{\top }A$ zero par map karta hai. SVD formula mein, ek zero singular value ke saath kya hota hai? ::: Woh ${\Sigma }^{+}$ mein zero rehta hai — us direction mein invert karne ke liye koi information nahi hoti. Fat-matrix answer ko infinitely many solutions mein unique kya banata hai? ::: Woh sabse chota hota hai, null space ke perpendicular (saari null content remove kar di gayi).

---

Connections

• Pseudoinverse — parent note; yeh page iska visual derivation hai.
• Orthogonal Projection — Steps 2–3, woh perpendicular jo "closest" define karta hai.
• Least Squares Regression — Step 4, tall-matrix application.
• Singular Value Decomposition — Step 7, universal engine.
• Four Fundamental Subspaces — column space, row space, null space poore mein.
• Eigenvalues and Eigenvectors — ${\sigma }_i^2$, $A^{\top }A$ ke eigenvalues ke roop mein.
• Matrix Inverse — woh special square case jahan yeh sab $A^{-1}$ mein collapse hota hai.