Critical points and saddle points — Hinglish notes

1.2.10 · AI-ML › Calculus & Optimization Basics

What Are Critical Points?

Gradient kyun vanish hota hai? Gradient $\nabla f$ steepest ascent ki direction mein point karta hai. Ek critical point par, koi aisi direction nahi hoti jo immediately $f$ ko increase ya decrease kare—function locally "flat" hota hai. Yeh humein nahi batata ki hum maximum, minimum, ya saddle par hain; yeh sirf kehta hai ki first-order behavior zero hai.

Derivation from first principles:

$f$ ke liye $x^{*}$ par local extremum hone ke liye, kisi bhi direction $v$ mein thoda move karne se $f$ first order tak change nahi hona chahiye: $f (x^{*} + ϵ v) \approx f (x^{*}) + ϵ \nabla f (x^{*})^{⊤} v$

Yeh extremum hone ke liye, linear term sabhi $v$ ke liye vanish hona chahiye, jo $\nabla f (x^{*}) = 0$ force karta hai.

Classifying Critical Points: The Hessian

Jab ek baar hum $\nabla f (x^{*}) = 0$ dhundh lete hain, tab hum Hessian matrix ke zariye second-order behavior examine karte hain.

Hessian kyun? $x^{*}$ ke aas-paas Taylor expansion deta hai: $f (x^{*} + d) \approx f (x^{*}) + \nabla f (x^{*})^{⊤} d + \frac{1}{2} d^{⊤} Hd$

Kyunki $\nabla f (x^{*}) = 0$ , behavior quadratic form $d^{⊤} Hd$ se govern hoti hai. Is form ka sign determine karta hai ki jab hum door move karte hain toh $f$ increase hoga ya decrease.

Eigenvalues kyun? $H$ ke eigenvalues principal axes ke along curvatures hain. Agar saare positive hain, toh function har direction mein upar curve karta hai (ek bowl). Agar mixed hain, toh kuch directions mein upar aur kuch mein neeche curve karta hai (ek saddle).

Figure — Critical points and saddle points

Saddle Points in Deep Learning

High-D mein saddles kyun dominate karte hain:
Ek random symmetric matrix ke liye (jo $H$ ka ek rough model hai), saare $n$ eigenvalues ke same sign hone ki probability $n$ ke saath exponentially shrink hoti hai. $R^{1000000}$ mein, local minimum ki chance astronomically small hai; saddles hi norm hain.

Optimization ke liye implication: Gradient descent saddles ke paas atak sakta hai jahan $∥\nabla f ∥ \approx 0$ ho lekin point minimum nahi hota. Adam, SGD with momentum, aur saddle-free Newton methods jaise algorithms noise add karke ya curvature information use karke saddles se escape karne mein help karte hain.

Worked example Example 3: Neural Network Loss

Ek do-layer network ko consider karo jiska loss $L (w)$ parameter space mein hai. Maano gradient descent $\nabla L (w^{*}) \approx 0$ dhundh leta hai aur $∥\nabla L ∥$ decrease hona band ho jaata hai.

Check karo ki saddle par atke hain ya nahi:

Hessian eigenvalues compute ya approximate karo (jaise Lanczos ya power iteration se)
Agar smallest eigenvalue $λ_{m i n} < 0$ hai, toh ek direction $v$ hai jahan loss decrease hoti hai: $v$ $λ_{m i n}$ ka eigenvector hai
Escape karne ke liye $w^{*} \to w^{*} + ϵ v$ perturb karo Yeh step kyun? Negative eigenvalue matlab loss surface direction $v$ mein neeche curve karti hai. Us direction mein move karne se loss decrease hoti hai, jo prove karta hai ki $w^{*}$ minimum nahi hai. Modern optimizers noise add karte hain (SGD ki minibatch variance) ya momentum use karte hain taaki saddles se naturally perturb ho sakein.

Common Mistakes

Active Recall

Recall Ek 12-saal ke bachche ko explain karo

Socho tum ek video game khel rahe ho jahan tum ek marble ho jo ek ऊबड़-खाबड़ surface par roll kar raha hai, aur tum sabse neechi valley dhundhna chahte ho. Ek critical point woh jagah hai jahan zameen bilkul flat ho—tum roll karna band kar do kyunki koi slope nahi hai. Lekin "flat" ka matlab hamesha valley ke bottom par hona nahi hota! Tum ho sakte ho:

Bowl ke bottom par (minimum)—tumne ek low point dhundh liya, yay!
Hill ke top par (maximum)—tum ek high point par ho, bura hai agar tum low chahte ho.
Saddle par (jaise ghode ki saddle)—beech mein flat, lekin agar aage roll karo toh neeche jaate ho, aur agar sideways roll karo toh upar jaate ho. Yeh figure out karne ke liye ki kaunsa hai, tum curvature check karte ho—kya zameen smile ki tarah upar curve karti hai (minimum), frown ki tarah neeche (maximum), ya mixed (saddle)? Math mein, Hessian matrix ek saath sari directions mein curvature batata hai.

Connections

Gradient Descent: Critical points woh jagahein hain jahan gradient descent converge hota hai (ya atak jaata hai)
Second-Order Optimization: Newton's method Hessian use karta hai minima ki taraf jump karne ke liye, saddles se bachte hue
Loss Landscape Visualization: Loss surface plots jaise tools deep nets mein saddles ki density reveal karte hain
Convex Optimization: Convex functions mein, har critical point global minimum hota hai—saddles impossible hain
Momentum and Adam: Yeh optimizers saddles se escape karne mein help karte hain velocity/adaptive learning rates ke zariye
Eigenvalues and Eigenvectors: Hessian ke eigenvalues/vectors principal curvatures describe karte hain
Taylor Series: Second-order Taylor expansion Hessian classification ko motivate karta hai

#flashcards/ai-ml

Ek function $f$ ka critical point kya hota hai? :: Ek aisa point $x^{*}$ jahan gradient vanish hota hai: $\nabla f (x^{*}) = 0$ . Saare partial derivatives zero hote hain.

Hessian matrix kya represent karta hai?

Second partial derivatives ka matrix

H_{ij} = \frac{\partial ^{2} f}{\partial x _{i} \partial x _{j}}

, jo kisi point par

f

ki curvature capture karta hai.

Hessian ke eigenvalues use karke critical point classify kaise karte hain?

Saare positive → local minimum; saare negative → local maximum; mixed signs → saddle point; kuch zero → degenerate (higher derivatives chahiye).

High-dimensional neural network optimization mein saddle points itne common kyun hote hain?

High dimensions mein, saare

n

eigenvalues ke same sign hone ki probability exponentially small hoti hai. Zyaadatar critical points ke mixed-sign eigenvalues hote hain, jo unhe saddles banate hain.

Critical point ke paas quadratic form approximation kya hoti hai?

f (x^{*} + d) \approx f (x^{*}) + \frac{1}{2} d^{⊤} Hd

, jahan gradient term

x^{*}

par vanish ho jaata hai.

Optimization mein saddle point se escape kaise karein?

Negative eigenvalue ke corresponding eigenvector ke along perturb karo, ya stochastic gradients aur momentum ke noise par rely karo taaki naturally escape ho sake.

Positive definite Hessian kya indicate karta hai?

Critical point ek local minimum hai—function har direction mein upar curve karta hai (saare eigenvalues

> 0

Indefinite Hessian kya indicate karta hai? :: Critical point ek saddle hai—function kuch directions mein upar aur kuch mein neeche curve karta hai (mixed-sign eigenvalues).