Singular Value Decomposition (SVD) — full derivation: Question bank

4.5.40 · D5 · Maths › Linear Algebra (Full) › Singular Value Decomposition (SVD) — full derivation

Har item parent note mein pehle se bani vocabulary pe rely karta hai: singular values $σ_{i}$ , right singular vectors $v_{i}$ ( $V$ ke columns), left singular vectors $u_{i}$ ( $U$ ke columns), $A$ ki shape wala matrix $Σ$ , aur gateway object $A^{⊤} A$ . Agar koi symbol unfamiliar lage, toh pehle parent derivation pe wapas jao.

True ya false — justify karo

Har matrix ka ek SVD hota hai.

True — derivation ne kabhi assume nahi kiya ki

A

square hai, invertible hai, ya symmetric hai; usne sirf yeh use kiya ki

A^{⊤} A

symmetric PSD hai, jo kisi bhi real

A

ke liye hold karta hai. Dekho Spectral Theorem.

Singular values negative ho sakte hain agar

A

mein negative entries hon.

False —

σ_{i} = λ_{i} (A^{⊤} A)

aur ek PSD matrix ke eigenvalues

λ_{i}

\geq 0

hote hain, isliye square root real aur non-negative hai by construction, chahe

A

ki entries kuch bhi hon.

Matrix

Σ

hamesha square hoti hai.

False —

Σ

ki shape $A$ jaisi hoti hai (

m \times n

); sirf jab

m = n

ho tab yeh square hoti hai. Warna isme zero-padding rows ya columns hoti hain.

Agar

A

symmetric hai, toh uske singular values uske eigenvalues ke barabar hote hain.

Generally False, sirf tab True jab

A

PSD bhi ho. Symmetric

A

ke liye,

σ_{i} = ∣ λ_{i} (A) ∣

, isliye ek negative eigenvalue

- 3

singular value

3

deta hai.

U

aur

V

uniquely

A

se determine hote hain.

False — repeated singular values apne shared subspace mein rotate karne ki freedom dete hain, aur har singular vector ka sign flip ho sakta hai (agar tum

v_{i}

flip karo toh

u_{i}

bhi flip karna padega taaki

A v_{i} = σ_{i} u_{i}

satisfy ho).

Σ

lekin unique hota hai.

U

ke columns usi space mein rehte hain jisme

V

ke columns rehte hain.

False —

u_{i} \in R^{m}

(output space),

v_{i} \in R^{n}

(input space). SVD do alag orthonormal frames use karta hai, yahi cheez ise eigen-decomposition se distinguish karti hai.

Zero singular values

A

mein kuch contribute nahi karte.

A

ko reconstruct karne ke liye True hai (unke outer products vanish ho jaate hain), lekin woh useless nahi hain — corresponding

v_{i}

A

ke null space ko span karte hain, isliye woh encode karte hain ki

A

kya destroy karta hai.

Ek orthogonal matrix ke saare singular values 1 ke barabar hote hain.

True — agar

Q^{⊤} Q = I

hai toh

Q^{⊤} Q

ka har eigenvalue

1

hai, isliye har

σ_{i} = 1 = 1

; ek orthogonal map bina stretch kiye rotate/reflect karta hai.

Nonzero singular values ki sankhya

A

ki rank ke barabar hoti hai.

True —

rank (A) = rank (A^{⊤} A)

A^{⊤} A

ke nonzero eigenvalues ki sankhya =

σ_{i} > 0

ki sankhya, jise derivation

r

kehti hai.

Error dhundo

U

paane ke liye, bas

A A^{⊤}

ke eigenvectors lo aur unhe

v_{i}

ke saath arbitrarily pair karo."

Pairing aur sign free nahi hai:

A A^{⊤}

ke eigenvectors sirf sign tak determine hote hain, isliye tum

- u_{i}

pe land kar sakte ho aur

A v_{i} = σ_{i} u_{i}

toot jaayegi. Fix: pehle

V

choose karo, phir

u_{i} = A v_{i} / σ_{i}

define karo.

u_{i} = A v_{i} / σ_{i}

U

ke saare

m

columns define karta hai."

Yeh sirf

u_{i}

σ_{i} > 0

ke liye define karta hai (yaani

i \leq r

ke liye). Remaining

u_{r + 1}, \dots, u_{m}

ko full orthonormal basis mein extend karke banana padta hai, jaise Gram–Schmidt ke zariye.

"Kyunki

σ_{1} \geq σ_{2} \geq \dots

, sabse bada singular value

A

ka sabse bada eigenvalue hai."

σ_{1}

A^{⊤} A

ka sabse bada eigenvalue hai, square-rooted —

A

ka eigenvalue nahi.

A = (0020)

ke liye,

A

ke saare eigenvalues

0

hain phir bhi

σ_{1} = 2

hai.

A = U Σ V^{⊤}

isliye

A^{- 1} = V Σ^{- 1} U^{⊤}

hamesha."

Sirf tab agar

A

square ho aur har

σ_{i} > 0

ho. Agar koi

σ_{i} = 0

hai toh

Σ

invertible nahi hai; sahi tool hai pseudoinverse

A^{+} = V Σ^{+} U^{⊤}

, sirf nonzero

σ_{i}

ko invert karke.

"Hum

A v_{i}

σ_{i}

se sirf normalisation convenience ke liye divide karte hain."

Yeh sirf normalise se zyada karta hai: yeh correct sign/direction pairing bhi fix karta hai taaki column identity

A V = U Σ

exactly hold kare. Isse skip karo aur poora assembly step fail ho jaata hai.

A^{⊤} A

aur

A A^{⊤}

ke bilkul alag eigenvalues hote hain."

Woh saare nonzero eigenvalues share karte hain (dono

σ_{i}^{2}

ke barabar hain); woh sirf is mein differ karte hain ki bade wale matrix mein kitne extra zero eigenvalues pad karte hain. Isliye ek

Σ

dono

U

aur

V

ke kaam aata hai.

Σ

diagonal hai matlab

A

diagonally act karta hai, isliye

A

diagonalizable hai."

Σ

do SVD frames mein diagonal hai, standard basis mein nahi, aur generally

U \neq = V

hota hai — yeh input aur output pe alag-alag basis change hai, jo eigen-diagonalization nahi hai. Ek defective (non-diagonalizable) matrix ka phir bhi SVD hota hai.

Why questions

Hum derivation

A^{⊤} A

se kyun shuru karte hain,

A

se seedha kyun nahi?

A

rectangular ho sakta hai jisme koi eigenvectors nahi hote, lekin

A^{⊤} A

hamesha symmetric PSD hota hai, isliye Spectral Theorem hume real non-negative eigenvalues ke saath ek orthonormal eigenbasis free mein de deta hai.

A^{⊤} A

ke eigenvalues non-negative kyun hone chahiye?

Kisi bhi

x

ke liye,

x^{⊤} A^{⊤} A x = ∥ A x ∥^{2} \geq 0

, isliye

A^{⊤} A

PSD hai; ek PSD matrix ke eigenvalues eigenvectors pe exactly yahi values hote hain, isliye

\geq 0

hain, jo hume real square roots lene deta hai.

SVD ko do orthogonal matrices kyun chahiye jab eigen-decomposition ko sirf ek chahiye?

Eigen-decomposition ek space ko khud map karta hai, isliye ek basis kaafi hai; ek general

A

input space

R^{n}

ko ek alag output space

R^{m}

mein map karta hai, isliye ise ek source frame (

V

) aur ek landing frame (

U

) chahiye.

u_{i}

automatically orthonormal kyun hote hain bina kisi extra kaam ke?

Kyunki

u_{i}^{⊤} u_{j} = \frac{1}{σ _{i} σ _{j}} v_{i}^{⊤} A^{⊤} A v_{j} = \frac{σ _{j}}{σ _{i}} δ_{ij}

v_{i}

(eigenvectors) ki orthonormality directly

u_{i}

mein transfer ho jaati hai.

SVD Principal Component Analysis (PCA) ke peeche natural tool kyun hai?

Centred data ke right singular vectors covariance matrix

\propto A^{⊤} A

ke eigenvectors hote hain, aur

σ_{i}^{2}

har direction mein captured variance ko rank karte hain — exactly principal components.

Sirf top

k

singular values rakhne se best rank-

k

approximation kyun milti hai?

Kyunki

u_{i}, v_{i}

frames orthonormal hain,

σ_{i}

contributions independent aur additive hain; sabse chhote ones ko drop karna sabse kam "stretch" remove karta hai, jo Eckart–Young theorem prove karta hai ki optimal hai — dekho Low-Rank Approximation.

Singular values ko

σ_{1} \geq σ_{2} \geq \dots

order mein kyun rakhte hain?

Ordering ek convention hai jo

Σ

ko unique banati hai, nonzero values ko pehle group karti hai (rank

r

ko null part se clearly alag karti hai), aur truncation hamesha sabse significant directions rakhe yeh ensure karti hai.

Edge cases

Zero matrix

A = 0

(size

m \times n

) ka SVD kya hoga?

Saare

σ_{i} = 0

hain isliye

Σ = 0

;

U

aur

V

koi bhi orthogonal matrices ho sakte hain, aur

U 0 V^{⊤} = 0

trivially hold karta hai. Rank

0

hai, har

v_{i}

null space span karta hai.

Jab

r < m

ho (output dimensions se kam nonzero singular values),

U

ke saath kya hota hai?

Formula

u_{i} = A v_{i} / σ_{i}

sirf

r

vectors produce karta hai; remaining

m - r

columns ko left null space

null (A^{⊤})

ke orthonormal basis ko extend karke freely choose kiya jaata hai.

Kya SVD complex eigenvalues wale matrix ke liye exist karta hai, jaise rotation?

Haan — SVD

A^{⊤} A

ke real eigenvalues se banta hai, kabhi

A

ke eigenvalues se nahi. Ek

9 0^{\circ}

rotation ke complex eigenvalues

\pm i

hain lekin singular values

1, 1

hain, jo correctly report karta hai "koi stretching nahi."

Ek tall, full-column-rank matrix (

m > n

r = n

) ke liye

Σ

kaisa dikhega?

m \times n

block jisme diagonal pe

n

positive singular values hain aur neeche

m - n

all-zero rows pad hain; koi zero singular values nahi hain, isliye saare

v_{i}

row space mein hain.

Agar ek singular value repeated ho, toh singular vectors mein kya freedom aata hai?

A^{⊤} A

ke shared eigenspace mein

v_{i}

ka koi bhi orthonormal rotation valid hai (aur

u_{i}

ka matching rotation force hota hai); decomposition ab unique nahi rehta lekin

Σ

phir bhi unique rehta hai.

1 \times 1

matrix

A = [a]

ka SVD kya hoga?

σ_{1} = ∣ a ∣

V = [1]

, aur

U = [sign (a)]

(agar

a = 0

toh

U = [1]

), isliye

U Σ V^{⊤} = ∣ a ∣ \cdot sign (a) = a

; sign

U

mein rehta hai, magnitude

Σ

mein.

Jab koi bhi singular value zero na ho toh

A

ka null space kya hai?

Yeh

{0}

hai: har

v_{i}

σ_{i} > 0

hai isliye

A v_{i} \neq = 0

, matlab

A

apne domain pe injective hai aur

r = n

; kuch bhi collapse nahi hota.

Recall Traps ka one-line summary

Singular values $A^{⊤} A$ se aate hain (real, $\geq 0$ , kabhi $A$ ke eigenvalues nahi); $Σ$ ki shape $A$ jaisi hai; $U$ aur $V$ alag spaces mein rehte hain aur $u_{i} = A v_{i} / σ_{i}$ se ek saath pin hote hain; zeros null space encode karte hain, garbage nahi.