Positive definite and semidefinite matrices — Hinglish notes

1.1.17 · AI-ML › Linear Algebra Essentials

WHY: hum kyun care karte hain? (80/20)

ML mein almost har "energy", "loss curvature", "covariance", ya "kernel" ek symmetric matrix hoti hai, aur hume hamesha jaanna hota hai: kya yeh cheez ek valid bowl-shaped cost ki tarah behave karti hai?

Optimization: agar kisi function ka Hessian PD hai ⇒ strict local minimum, toh gradient descent achhe se converge karta hai.
Covariance matrices hamesha PSD hoti hain (variance negative nahi ho sakta).
Kernels (SVMs, Gaussian processes) PSD hone chahiye (Mercer's condition) taaki ek valid inner product ke corresponding ho sakein.
Cholesky / $A x = b$ solve karna fast aur stable hota hai exactly jab $A$ PD ho.

Toh yeh ek akela concept quietly loss landscapes, statistics, aur kernel methods ko power karta hai.

WHAT: definitions kya hain?

Hum likhte hain $A ≻ 0$ (PD) aur $A ⪰ 0$ (PSD). Negative (semi)definite sign flip kar deta hai; indefinite ka matlab hai $x^{⊤} A x$ dono signs leta hai (saddle).

HOW: eigenvalue test scratch se derive karna

Hum claim karte hain: symmetric $A$ ke liye, " $x^{⊤} A x > 0$ for all $x \neq = 0$ " ⟺ "all eigenvalues $> 0$ ".

Step 1 — Spectral theorem. Ek real symmetric matrix ko likha ja sakta hai $A = Q Λ Q^{⊤}, Q^{⊤} Q = I, Λ = diag (λ_{1}, \dots, λ_{n}) .$ Yeh step kyun? Symmetric matrices ke real eigenvalues aur ek orthonormal eigenbasis hota hai — yahi woh key structural fact hai jo hume coordinates change karne deta hai bina distortion ke.

Step 2 — Change of variables. Maano $y = Q^{⊤} x$ . Kyunki $Q$ orthogonal hai, $x \neq = 0 ⟺ y \neq = 0$ . Tab $x^{⊤} A x = x^{⊤} Q Λ Q^{⊤} x = (Q^{⊤} x)^{⊤} Λ (Q^{⊤} x) = y^{⊤} Λ y .$ Yeh step kyun? Eigenbasis mein rotate karne se matrix diagonal ho jaati hai — "cross terms" gayab ho jaate hain.

Step 3 — Diagonal form. $y^{⊤} Λ y = \sum_{i = 1}^{n} λ_{i} y_{i}^{2} .$ Yeh step kyun? Ek diagonal quadratic form sirf squares ka weighted sum hota hai — padhna bilkul aasaan hai.

Step 4 — Conclude. Har $y_{i}^{2} \geq 0$ .

Agar sab $λ_{i} > 0$ hain, toh sum $> 0$ hoga jab bhi koi $y_{i} \neq = 0$ ho ⇒ PD.
Agar koi $λ_{j} \leq 0$ ho, toh $y = e_{j}$ lo (yani $x = Q e_{j}$ , $j$ -th eigenvector): sum $λ_{j} \leq 0$ ke barabar hoga ⇒ PD nahi.

$■$ Wohi argument $\geq$ ke saath PSD ⟺ $λ_{i} \geq 0$ result deta hai.

Figure — Positive definite and semidefinite matrices

Formula Symmetric

$A$ ke liye practical tests

Property	Test
PD	all $λ_{i} > 0$ • all leading principal minors $> 0$ (Sylvester) • $A = R^{⊤} R$ with $R$ full column rank • Cholesky exists
PSD	all $λ_{i} \geq 0$ • all principal minors $\geq 0$ • $A = B^{⊤} B$ for some $B$

Worked examples

Common mistakes (steel-manned)

Recall Feynman: 12-saal ke bachche ko samjhao

Ek skateboard bowl imagine karo. Koi bhi marble kahan bhi daalo: agar ground har direction mein upar curve kare, toh marble hamesha neeche wapis aayega — yeh positive definite bowl hai. Agar bowl ka koi hissa flat gutter jaisa ho, toh marble us gutter mein ruk sakta hai (neeche nahi girta par girta bhi nahi) — yeh semidefinite hai. Agar yeh mountain-pass saddle jaisa ho, toh ek taraf se girta hai aur doosri taraf se chadh jaata hai — yeh indefinite hai. " $x^{⊤} A x$ " number bas ground ki height hai jab tum direction $x$ mein chalte ho; positive definite ka matlab hai woh height hamesha positive hai siwaay center ke.

Recall flashcards

Ek positive definite symmetric matrix ko kya define karta hai?

x^{⊤} A x > 0

for all

x \neq = 0

; equivalently all eigenvalues

> 0

PD aur PSD mein ek word mein kya difference hai?

Strict (

> 0

, koi zero eigenvalue nahi) vs allows-flat (

\geq 0

, zero eigenvalues allowed).

Definiteness test karne se pehle

A

symmetric kyun maanna zaroori hai?

Sirf symmetric part hi

x^{⊤} A x

ko affect karta hai, kyunki antisymmetric part zero contribute karta hai.

PD ke liye Sylvester's criterion kya hai?

Saare leading principal minors positive hain.

A = B^{⊤} B

PSD kyun guarantee karta hai?

x^{⊤} B^{⊤} B x = ∥ B x ∥^{2} \geq 0

hamesha.

Hessian ka PD hona kab useful hai?

Critical point par yeh strict local minimum certify karta hai.

Kya covariance matrix hamesha PD hoti hai?

Nahi — hamesha PSD hoti hai; PD tabhi jab data kisi lower-dim subspace tak confined na ho.

Counterexample: positive entries par PD nahi?

[1221]

, eigenvalues

3, - 1

(indefinite).

Kya

det A > 0

PD imply karta hai?

Nahi;

- I

det = 1 > 0

hai par yeh negative definite hai.

Connections

Eigenvalues and Eigenvectors — definitive test yahan rehta hai.
Spectral Theorem — derivation mein use ki gayi orthonormal diagonalization provide karta hai.
Cholesky Decomposition — exist karta hai iff PD; fast solver.
Covariance Matrix — statistics mein canonical PSD object.
Kernels and Mercer's Theorem — valid kernels ⟺ PSD Gram matrices.
Hessian and Convexity — PD Hessian ⇒ strictly convex ⇒ unique minimum.
Quadratic Forms — $x^{⊤} A x$ direction ka scalar function.