The Hessian matrix

1.2.6 · AI-ML › Calculus & Optimization Basics

Hessian KYA hai?

$n = 2$ ke liye variables $x, y$ ke saath:

H = \frac{\partial ^{2} f}{\partial x ^{2}} \frac{\partial ^{2} f}{\partial y \partial x} \frac{\partial ^{2} f}{\partial x \partial y} \frac{\partial ^{2} f}{\partial y ^{2}}

Order kyun matter nahi karta? Dono mixed partials surface ka wahi "twist" measure karte hain — $x$ -direction mein slope $y$ mein nudge ke response mein kitni badlati hai, ye $y$ mein slope ka $x$ mein nudge ke response mein badalna ke barabar hai. Geometrically ye ek hi warping hai, kisi bhi angle se measure ki gayi.

Ye symmetry ek gift hai: ek real symmetric matrix ke real eigenvalues aur ek orthonormal eigenbasis hota hai (spectral theorem) — exactly wahi jo humein critical points classify karne ke liye chahiye.

Hessian kaise aata hai: 2nd-order Taylor expansion

Step 1 — 1D reminder. $a$ ke paas: $f (a + h) = f (a) + f^{'} (a) h + \frac{1}{2} f^{''} (a) h^{2} + \dots$ Ye step kyun? Ye woh reference hai jise hum generalize karenge; $\frac{1}{2} f^{''} (a) h^{2}$ term curvature contribution hai.

Step 2 — Ek line tak restrict karo. $R^{n}$ mein, ek point $x$ aur ek direction $v$ chuno, aur $g (t) = f (x + t v)$ define karo, jo $t$ ka ek 1D function hai. Ye step kyun? Kisi bhi straight path par multivariable behaviour ek 1D function hota hai — toh hum 1D Taylor reuse kar sakte hain.

Step 3 — $g$ ko chain rule se differentiate karo. $g^{'} (t) = \nabla f (x + t v)^{⊤} v, g^{''} (t) = v^{⊤} H (x + t v) v$ Ye step kyun? $g^{'} (t)$ directional derivative hai. Dobara differentiate karne par, chain rule ek gradient ka doosra gradient neeche le aata hai — aur woh precisely Hessian hai jo $v$ 's ke beech sandwich hua hai.

Step 4 — $t = 0$ par 1D Taylor mein plug karo, phir $t = 1$ , $v = Δ x$ set karo: $f (x + Δ x) \approx f (x) + \nabla f (x)^{⊤} Δ x + \frac{1}{2} Δ x^{⊤} H (x) Δ x$

Ye kyun matter karta hai: aakhri term $\frac{1}{2} Δ x^{⊤} H Δ x$ ek quadratic form hai. Har direction mein iska sign tumhe surface ki shape batata hai.

Hessian se critical points classify karna

Critical point $\nabla f = 0$ par, linear term vanish ho jaata hai aur Hessian akela local shape decide karta hai.

Eigenvalues kyun? Eigenbasis mein, quadratic form ban jaata hai $\frac{1}{2} \sum_{i} λ_{i} u_{i}^{2}$ . Har eigenvalue apne eigenvector ke along curvature hai. Har direction mein positive ⇒ har rasta upar curve karta hai ⇒ minimum.

Worked examples

AI/ML kyun care karta hai

Intuition Optimization curvature use karta hai

Newton's method update karta hai $x \leftarrow x - H^{- 1} \nabla f$ : ye Hessian use karke local quadratic model ke minimum tak seedha jump karta hai — plain gradient descent se kahin zyada faster minimum ke paas.
$H$ ke eigenvalues loss landscape describe karte hain: bada ratio $λ_{m a x} / λ_{m i n}$ (condition number) ka matlab hai ek lamba patla valley jahan gradient descent zig-zag karta hai.
Hessian bahut bada hota hai ( $n$ parameters ke liye $n \times n$ ), isliye deep learning saste Hessian approximations use karta hai jaise L-BFGS (jo gradient differences se ek low-rank estimate banata hai). Note: Adam jaese optimizers Hessian approximations nahi hain — ye har parameter ki step size ko squared gradients (first-order statistics) se adapt karte hain, jo sasta hai lekin directly curvature estimate nahi karta.

Common mistakes

Flashcards

Hessian matrix kya collect karta hai?

Ek scalar function ke saare second-order partial derivatives,

H_{ij} = \partial^{2} f / \partial x_{i} \partial x_{j}

Hessian symmetric kyun hai?

Schwarz's theorem ki wajah se: agar second partials continuous hain, toh mixed partials commute karte hain, toh

H = H^{⊤}

x

ke baare mein

f

ka 2nd-order Taylor expansion kya hai?

f (x + Δ x) \approx f (x) + \nabla f^{⊤} Δ x + \frac{1}{2} Δ x^{⊤} H Δ x

Critical point par,

H

ke sab-positive eigenvalues ka kya matlab hai?

Local minimum (positive definite Hessian).

Critical point par, mixed-sign eigenvalues ka kya matlab hai?

Ek saddle point.

2×2 Hessian ke liye,

det H < 0

kya imply karta hai?

Ek saddle point (opposite sign ke eigenvalues).

2×2 mein

det H > 0

ke saath, min aur max mein fark kaise karo?

Top-left entry

a

(ya trace) check karo:

a > 0

→min,

a < 0

→max.

Newton's method

H^{- 1} \nabla f

kyun use karta hai?

Ye local quadratic Taylor model ko exactly ek step mein minimize karta hai, curvature use karke step ko scale/rotate karne ke liye.

H

ka bada condition number

λ_{m a x} / λ_{m i n}

kya indicate karta hai?

Ek ill-conditioned, lamba patla valley jahan gradient descent dhheere zig-zag karta hai.

Second-derivative test kab inconclusive hoti hai?

Jab

H

ka koi eigenvalue zero ho (ek flat direction).

Kya Adam Hessian approximate karta hai?

Nahi — Adam squared gradients (first-order statistics) se per-parameter steps adapt karta hai; ye curvature estimate nahi karta. L-BFGS ek true (quasi-Newton) Hessian approximation hai.

Recall Feynman: ek 12-saal ke bachche ko explain karo

Socho tum ek pahadi landscape par hiking kar rahe ho. Gradient ek compass ki tarah hai jo hamesha seedha uphill point karta hai — ye tumhe batata hai kahan chalna hai. Lekin ye tumhare aas-paas ki shape nahi batata: kya tum valley ke bottom mein ho, pahadi ki choti par ho, ya ghode ki saddle par jahan ek taraf upar jaata hai aur doosri taraf neeche? Hessian ek chhota sa "bend-o-meter" hai: har direction ke liye ye batata hai ki zameen kitni tezi se curve kar rahi hai. Agar ye har direction mein upar curve kare, toh tum ek bowl ke bottom mein ho (minimum). Agar ye har jagah neeche curve kare, toh tum summit par ho. Agar ye ek taraf upar aur doosri taraf neeche curve kare — woh saddle hai. Computers is bend info ko use karke fastest valley mein pahunchte hain.

Connections

The gradient vector — first-order info; Hessian iska derivative hai.
Taylor series (multivariable) — Hessian quadratic term hai.
Eigenvalues and eigenvectors — definiteness classify karne ke liye use hote hain.
Positive definite matrices — min-detecting condition.
Newton's method — optimization steps ke liye $H^{- 1}$ use karta hai.
Gradient descent — iska speed Hessian ke condition number se govern hota hai.
Convex functions — convex ⇔ Hessian har jagah positive semi-definite.