4.5.40 · D1Singular Value Decomposition (SVD) — full derivation
Koi bhi matrix, chahe kitni bhi "ugly" ho, secretly sirf rotate → stretch → rotate hai. "A = U Σ V ⊤ rotations (U V pure stretch (Σ
Yeh Singular Value Decomposition (SVD) — full derivation ka prerequisite page hai. Pehle ise padho. Hum har woh symbol name karenge jo parent note tumhare upar fire karta hai, uske peeche ki picture draw karenge, aur explain karenge kyun derivation uske bina aage nahi badh sakti .
Poore page mein, do families of special vectors par dhyan rakho: v 1 , … , v n right singular vectors , V u 1 , … , u m left singular vectors , U "woh special input/output directions jo SVD produce karega" samjho.
Definition Matrix = ek machine jo arrows move karti haiEk matrix A function jo ek vector khaata hai aur ek vector ugalta hai . Isko ek arrow x A x
Simple words mein: rows aur columns of numbers.The picture: space ka ek transformation. Pehle arrows ka ek grid; wahi arrows baad mein move ho gaye.Topic ko iske kya zaroorat hai: SVD ek claim hai ki yeh machine space ke saath kya karti hai — yeh kehta hai ki messy motion hamesha "spin, stretch, spin" hoti hai. Toh pehle humein matrix ko motion ki tarah dekhna hoga, spreadsheet ki tarah nahi.
Figure s01 (below): wahi arrows ka grid do baar dikhaya gaya — left mein untouched, aur right mein machine A A space ka motion hai, spreadsheet nahi.
Shape notation. Hum likhte hain A ∈ R m × n
A ∈ R m × n
R real numbers (ordinary decimals jaise 3 , − 1.5 , 2 m × n m n The picture: machine un arrows ko leta hai jo n m n m Why: SVD ka poora point yahi hai ki m n alag-alag rooms hain, isliye humein do sets of directions chahiye, ek har room ke liye.
Ek vector x x = ( x 1 , x 2 , … , x n ) ⊤ ⊤ transpose ) yahan bas matlab hai "is row ko ek column ki tarah khada kar do".
The picture: space mein ek arrow jiske paas length aur direction hai.Topic ko iske kya zaroorat hai: SVD poori tarah ek story hai ki A
A ⊤ Transpose ek matrix ko uske diagonal ke across flip karta hai: row i i A m × n A ⊤ n × m
The picture: grid ko ek mirror ke saamne rakh do jo top-left se bottom-right diagonal ke along rakha gaya ho.Topic ko iske kya zaroorat hai: poori derivation object A ⊤ A
A ⊤ A
Definition Dot product aur lengthDo vectors ka dot product hai x ⊤ y = x 1 y 1 + x 2 y 2 + ⋯ + x n y n squared length deta hai:
∥ x ∥ 2 = x ⊤ x = x 1 2 + ⋯ + x n 2 . ∥ x ∥ length (norm) hai — Pythagoras ke hisaab se origin se tip tak straight-line distance.
Topic ko iske kya zaroorat hai: proof line ∥ A v ∥ 2 = ( A v ) ⊤ ( A v ) = v ⊤ A ⊤ A v kuch nahi hai sirf dot-product-is-squared-length trick ke. Har singular value σ ∥ A v ∥ v
Figure s02 (below): ek arrow x = ( 3 , 2 ) ∥ x ∥ x x ⊤ x = x 1 2 + x 2 2 Pythagoras hi hai — squared length.
Intuition Kyun dot product angle bhi measure karta haix ⊤ y = ∥ x ∥ ∥ y ∥ cos θ θ
Agar x ⊤ y = 0 perpendicular hain (cos 90° = 0
x ⊤ x = ∥ x ∥ 2 0 cos 0 = 1
Definition Orthonormal setVectors ka ek set orthonormal hota hai agar har vector ki length 1 normal ) aur har pair perpendicular ho (ortho ). Compactly:
v i ⊤ v j = δ ij .
The picture: unit arrows ka ek set jo sab right angles par hain — axes ka ek clean, un-skewed set.Topic ko iske kya zaroorat hai: SVD jo do frames produce karta hai (v u U V rotations banata hai sloppy skews ki jagah.
Definition Identity matrixI I I x = x 1 0 I n n × n
Definition Orthogonal matrixEk square matrix Q orthogonal hai agar iske columns ek orthonormal set form karte hain, equivalently Q ⊤ Q = I
The picture: Q rigid motion hai — ek rotation ya reflection. Yeh kabhi stretch nahi karta, kabhi squish nahi karta; lengths aur angles preserve hoti hain. Isko ek square grid do, usi size ka ek rotated square grid milega.Topic ko iske kya zaroorat hai: A = U Σ V ⊤ U V exactly yahi hain. V U
Figure s03 (below): unit axes pehle (left) aur ek orthogonal matrix Q rotate karte hain. Yeh U V
Q ⊤ Q = I i j Q ⊤ Q ( i , j ) Q ⊤ Q = I δ ii = 1 δ ij = 0 Q ⊤ Q = I Q ⊤ = Q − 1 undo karna free hai.
Pehle, woh word jis par hum aasra lene wale hain:
r σ r A rank r genuinely independent output directions ki sankhya hai jo machine produce kar sakti hai — iske output image ki dimension. Poori picture aur "why" §4 mein; abhi ke liye bas r
σ 1 ≥ σ 2 ≥ ⋯ ≥ σ r > 0 = σ r + 1 = …
Yahan σ r nonzero stretch hai — pehle r r
Vectors ke ek set ka span har woh arrow hai jo tum unhe scale aur add karke bana sakte ho. Ek plane mein do non-parallel arrows poore plane ko span karte hain; do parallel wale sirf ek line span karte hain.
The picture: arrows ko combine karke "reach ki ja sakne wali" poori region — ek line, ek plane, ya poora space.Topic ko iske kya zaroorat hai: column space, row space aur null space sab chosen vectors ke spans hain; SVD ka payoff yeh hai ki yeh tumhe har ek ke liye clean orthonormal spanning sets de deta hai.
r A
The picture: machine ko poora input space do; output space mein jo shadow padta hai woh r Why: parent singular values ko σ 1 … σ r > 0 σ r + 1 ⋯ = 0
Four Fundamental Subspaces woh chaar rooms hain jismein har matrix rehti hai. SVD tumhe ek saath chaalon ke liye orthonormal basis deta hai — isliye hum inhe yahan name karte hain.
A x A x = 0 origin par crush kar deti hai .
The picture: input directions jo ek single point par flatten ho jaate hain.Topic ko iske kya zaroorat hai: SVD mein yeh crushed directions exactly woh right singular vectors v i σ i = 0
A A rows ka span (equivalently A ⊤ crush nahi hoti .
The picture: input space ka r Topic ko iske kya zaroorat hai: right singular vectors v 1 , … , v r σ i > 0
A y A ⊤ y = 0 A reach nahi kar sakta .
The picture: output space ka woh hissa jo khali reh jaata hai, column space ke perpendicular.Topic ko iske kya zaroorat hai: bache hue left singular vectors u r + 1 , … , u m U
Ab hum precisely define kar sakte hain jo top par promise kiya tha.
Definition Right singular vectorsv i V Right singular vectors v 1 , … , v n input space R n V = [ v 1 ∣ ⋯ ∣ v n ]
The picture: clean input frame jo SVD choose karta hai — woh axes jinke along A Topic ko iske kya zaroorat hai: V ⊤ pehla rotation hai; yeh tumhare input ko is special frame par spin karta hai.
Definition Left singular vectorsu i U Left singular vectors u 1 , … , u m output space R m U = [ u 1 ∣ ⋯ ∣ u m ]
The picture: clean output frame jahan SVD land karta hai — A v i σ i u i Topic ko iske kya zaroorat hai: U doosra rotation hai; Σ
Definition Eigenvector & eigenvalueEk square matrix M eigenvector v direction machine nahi badalta — woh sirf ise scale karta hai: M v = λ v λ eigenvalue hai.
The picture: M Why: derivation ka engine eigenproblem A ⊤ A v i = σ i 2 v i A ⊤ A hi right singular vectors hain; eigenvalues hi squared singular values hain.
Poori detail Eigenvalues and Eigenvectors mein hai.
Definition Symmetric matrixM symmetric hai agar M ⊤ = M A ⊤ A ( A ⊤ A ) ⊤ = A ⊤ A
Definition Positive semi-definite (PSD)M PSD hai agar har x x ⊤ M x ≥ 0 M = A ⊤ A x ⊤ A ⊤ A x = ∥ A x ∥ 2 ≥ 0
Orthonormal u i extend karne ke liye, parent Gram-Schmidt Process invoke karta hai — kisi bhi spanning set ko orthonormal mein badalne ki standard machine.
Symbol
Bol ke padho
Picture
SVD mein role
A the matrix
space ka motion
woh cheez jo hum decompose karte hain
A ⊤ transpose
diagonal ke across mirror
A ⊤ A
x ⊤ y dot product
length & angle
σ i = ∥ A v i ∥
δ ij Kronecker delta
identity grid
orthonormality encode karta hai
U , V orthogonal
rotations
do spins (left/right singular vectors)
Σ diagonal
pure stretch
the singular values
λ i , v i eigen-pair
invariant direction
σ i 2
r rank
output dimension
nonzero σ i
Orthonormal sets and delta
Orthogonal matrices U and V
Symmetric matrix A-transpose-A
Eigenvalues and eigenvectors
SVD A = U Sigma V-transpose
Khud ko test karo — answer karne ke baad hi reveal karo.
Main A ∈ R m × n Input R n n R m m
Main ek transpose A ⊤ Main diagonal ke across flip karo: row i i m × n n × m
Main squared length ko dot product ki tarah likh sakta hoon ∥ x ∥ 2 = x ⊤ x = x 1 2 + ⋯ + x n 2
Main jaanta hoon ki x ⊤ y = 0 Dono arrows perpendicular hain.
Main bata sakta hoon δ ij 1 i = j 0
Main ek orthogonal matrix do tarah se define kar sakta hoon Columns orthonormal hain; equivalently Q ⊤ Q = I Q ⊤ = Q − 1
Main describe kar sakta hoon ki ek diagonal matrix space ke saath kya karta hai Har coordinate axis ko independently stretch karta hai, koi rotation nahi.
Main eigenvalue equation likh sakta hoon M v = λ v λ
Main jaanta hoon kyun A ⊤ A ( A ⊤ A ) ⊤ = A ⊤ A x ⊤ A ⊤ A x = ∥ A x ∥ 2 ≥ 0
Main jaanta hoon ki (real) Spectral Theorem mujhe kya deta hai Ek real symmetric matrix ke paas real eigenvalues ke saath orthonormal eigenbasis hota hai.
Main rank aur null space define kar sakta hoon Rank = output image ki dimension; null space = woh inputs jo 0
Main chaar fundamental subspaces ke naam le sakta hoon Row space aur null space (input side), column space aur left null space (output side).
Taiyaar ho? Singular Value Decomposition (SVD) — full derivation par jaao. SVD samajhne ke baad related destinations: Low-Rank Approximation , Principal Component Analysis (PCA) , Moore-Penrose Pseudoinverse .
