First-order optimality conditions — gradient = 0: Question bank

4.10.18 · D5 · Maths › Advanced Topics (Elite Level) › First-order optimality conditions — gradient = 0

Hamare kaam aane wale symbols ki reminder (sab parent note mein define hain, yahan dobaara likhe hain taaki yeh page akela bhi kaam kare):

$\nabla f$ — gradient, saare partial derivatives ka vector; yeh steepest increase ki taraf point karta hai. Dekho Gradient and directional derivatives.
Ek stationary point woh koi bhi point hai jahan $\nabla f = 0$ ho.
FONC = First-Order Necessary Condition: interior optimum par, $\nabla f = 0$ .
Hessian $H$ second partials ki matrix hai; yeh classify karta hai — dekho Hessian matrix and second-order conditions.

Definition Notation jo hum is page par actually use karenge

$∥ v ∥$ — vector $v$ ka Euclidean norm (length); 2-D mein, $∥ v ∥ = v_{1}^{2} + v_{2}^{2}$ . Vector ko uski length se divide karne par unit vector milta hai (length $1$ ), aur yahi "direction" hota hai.
$e_{i}$ — axis $i$ ke along unit coordinate vector: isme slot $i$ mein $1$ aur baaki sab jagah $0$ hota hai. Jaise $e_{1} = (1, 0)$ $x$ ke along point karta hai, $e_{2} = (0, 1)$ $y$ ke along. " $e_{i}$ ke along step karna" matlab sirf us ek coordinate mein move karna.
$D_{u} f (x)$ — directional derivative: $f$ kitni tezi se change hoti hai jab tum $x$ se unit direction $u$ mein step karte ho. Yeh dot product $\nabla f (x) \cdot u$ ke barabar hota hai (proof Gradient and directional derivatives par hai). Negative value matlab $f$ $u$ ke along decrease ho rahi hai; zero matlab us direction mein flat hai.

Figure 1 — ±h trapping argument (interior kyun matter karta hai). Interior minimum par tum dono taraf — left aur right — step kar sakte ho; right slope $\geq 0$ hai, left slope $\leq 0$ hai, aur differentiability dono ko $0$ par squeeze kar deti hai.

Figure — First-order optimality conditions — gradient = 0

Figure 2 — teen flat shapes. Minimum (bowl), maximum (dome) aur saddle teeno mein $\nabla f = 0$ hota hai; sirf curvature unhe alag karta hai.

Figure 3 — Hessian eigenvalue sign chart. Do eigenvalues ke signs kaise min / max / saddle / inconclusive se map karte hain.

Figure 4 — boundary optimum. Closed domain par sabse neeche wala point wall par baith sakta hai, jahan $\nabla f$ phir bhi outward point karta hai (non-zero) kyunki bahar step karna forbidden hai.

True or false — justify

Domain ke interior mein har local minimum par

\nabla f = 0

hota hai.

True — yahi exactly FONC hai; agar

\nabla f

ka koi bhi component non-zero hota toh tum us coordinate mein step karke

f

ko lower kar sakte the, jo "minimum" ko contradict karta (Figure 1).

Har woh point jahan

\nabla f = 0

ho, local minimum hota hai.

False — flatness maxima aur saddles par bhi hoti hai (Figure 2); FONC sirf candidates ko shortlist karta hai, winner declare nahi karta.

Agar

x

par

\nabla f \neq = 0

hai, toh

x

local max ya min nahi ho sakta.

True — non-zero gradient ek strict downhill direction

u = - \nabla f / ∥ \nabla f ∥

deta hai (yahan

∥ \cdot ∥

Euclidean length hai) aur ek strict uphill direction

- u

, toh tum

f

ko badhaa aur ghata dono kar sakte ho; koi bhi extremum possible nahi.

f (x) = x^{3}

x = 0

par minimum hai kyunki

f^{'} (0) = 0

False —

f^{'} (0) = 0

sirf

0

ko stationary banata hai, lekin

f

right taraf badhti hai aur left taraf girती hai, toh

0

ek inflection hai, na max na min.

Closed domain par minimum ko

\nabla f = 0

satisfy karna zaroor hai.

False — agar minimum boundary par baitha ho, toh gradient outward point karke non-zero reh sakta hai (Figure 4); tum domain ke bahar step nahi kar sakte, isliye FONC ka "both directions" wala argument toot jaata hai. Wahan Lagrange multipliers/KKT use karo.

Agar stationary point par

H

(Hessian) positive definite hai, toh point strict local minimum hai.

True — positive definite matlab

f

har direction mein upward curve karti hai (Figure 3 ka top-right cell), toh nearby saari

f

-values

f (x^{*})

se zyaada hoti hain; yeh second-order sufficient condition hai.

Stationary point par positive definite Hessian global minimum guarantee karta hai.

False in general — yeh sirf local minimum guarantee karta hai; globality ke liye puri domain par

f

ki convexity jaisi extra structure chahiye.

Convex

f

ke liye, har stationary point global minimum hota hai.

True — convexity matlab graph kisi bhi tangent plane ke neeche kabhi nahi jaata; kyunki stationary tangent plane flat hoti hai, koi bhi point

x^{*}

se neeche nahi ho sakta, toh yeh global hai.

Stationary point par har direction mein directional derivative

D_{u} f (x^{*}) = \nabla f (x^{*}) \cdot u

zero hoti hai.

True — agar

\nabla f (x^{*}) = 0

hai toh

0 \cdot u = 0

sab unit

u

ke liye; yeh "kisi bhi direction mein slope nahi" precisely flatness ka matlab hai.

Saddle point par non-zero gradient hota hai kyunki surface clearly tilted hoti hai.

False — saddle par

\nabla f = 0

hota hai; surface ek axis ke along rise karti hai aur doosre ke along fall (Figure 2, right), lekin exact centre par har first-order slope cancel hokar zero ho jaata hai.

Spot the error

f^{'} (x^{*}) \geq 0

right se aur

f^{'} (x^{*}) \leq 0

left se, toh

f^{'} (x^{*})

beech mein koi bhi number ho sakta hai." — galti kahan hai?

Differentiability left aur right limits ko same number hone par force karti hai (Figure 1); ek single value jo dono

\leq 0

aur

\geq 0

ho, sirf

0

ho sakti hai. Koi "beech mein" nahi hota.

"Humne

\nabla f = 0

find kar liya, toh kaam khatam — yahi woh minimum hai jo hume chahiye tha." — kya missing hai?

Classification step;

\nabla f = 0

necessary hai but sufficient nahi, toh minimum claim karne se pehle Hessian test run karna padega (Figure 3).

"Gradient minimum ki taraf downhill point karta hai, toh gradient descent

+ \nabla f

ke along move karta hai." — ise theek karo.

Gradient steepest increase ki taraf point karta hai; descend karne ke liye tum

- \nabla f

ke along move karte ho. Dekho Gradient descent.

"Boundary point par derivative zero honi chahiye kyunki yeh phir bhi optimum hai." — yeh kyun galat hai?

1-D proof ko dono

+ h

aur

- h

mein step karna padta hai; boundary par un directions mein se ek domain ke bahar jaati hai, toh inequality sirf ek side par hold karti hai aur derivative ka vanish karna zaruri nahi (Figure 4).

f (x, y) = x^{2} - y^{2}

mein origin par

\nabla f = 0

hai aur

H

mein ek positive aur ek negative eigenvalue hai, toh yeh

x

mein minimum aur

y

mein maximum hai — ise mixed minimum kaho." — terminology correct karo.

Mixed-sign eigenvalues Hessian ko indefinite banate hain (Figure 3 ka bottom-left cell), jo saddle point ki definition hai; yeh na min hai na max, koi "mixed minimum" nahi.

"Kyunki

g (t) = f (x^{*} + t e_{i})

mein har axis vector

e_{i}

ke liye

g^{'} (0) = 0

hai, toh point optimum hai." — overreach pakdo.

Axes

e_{i}

ke along slopes ka vanish karna sirf

\nabla f = 0

(stationarity) deta hai, optimality nahi; ek saddle mein bhi dono axes ke along zero slope hota hai phir bhi woh koi optimum nahi hai (Figure 2, right).

f

x^{*}

par differentiable nahi hai lekin minimum hai, toh FONC violate ho raha hai." — kya FONC actually break ho raha hai?

Nahi — FONC differentiable

f

ke liye stated hai. Agar

\nabla f

exist nahi karta, FONC simply apply nahi hota; corner (jaise

∣ x ∣

0

par) theorem ke scope ke bahar ek valid non-smooth optimum hai.

Why questions

FONC hold karne ke liye point interior kyun hona chahiye?

Kyunki proof dono directions

\pm h

mein step karta hai taaki

f^{'}

\leq 0

aur

\geq 0

ke beech trap kar sake (Figure 1); sirf ek interior point ke paas domain ke andar dono taraf move karne ki jagah hoti hai.

Non-zero gradient optimum ko kyun rule out karta hai, ek sentence mein?

Kyunki

D_{u} f = \nabla f \cdot u

u = - \nabla f / ∥ \nabla f ∥

ke liye strictly negative hota hai, toh

f

u

ke along strictly decrease karti hai — tum hamesha better kar sakte ho, isliye yeh minimum nahi hai.

Hum

D_{u} f = \nabla f \cdot u

mein kisi aur combination ki jagah dot product kyun use karte hain?

Dot product measure karta hai ki gradient ki steepest-increase direction kitni

u

ke saath align karti hai; yeh exactly

u

ke along ek unit step par

f

mein first-order change hai, jo ek directional slope ka matlab hota hai.

FONC maxima aur saddles ko bhi mins ke saath kyun dhundh leta hai?

Teeno first order par flat hote hain — tangent plane horizontal hoti hai (Figure 2) — aur gradient sirf first-order (slope) information dekhta hai, toh woh curving-up aur curving-down ya mixed mein distinguish nahi kar sakta.

Second-order (Hessian) test ki zaroorat kyun padti hai?

Gradient slope encode karta hai lekin curvature nahi; min vs max vs saddle decide karne ke liye yeh chahiye ki surface kaise bend karta hai, jo Hessian mein collected second derivatives mein rehta hai (Figure 3).

Convexity "stationary" ko "global minimum" tak kyun upgrade kar deti hai?

Ek convex function apne har tangent plane ke upar hamesha rehti hai;

x^{*}

par ek flat tangent phir poore graph ko neeche se

f (x^{*})

se bound karta hai, toh koi bhi point kahin bhi neeche nahi hota.

Edge cases

f

har jagah constant hai. Stationary points kahan hain aur woh kya hain?

\nabla f = 0

har point par hai; har ek simultaneously ek (weak) local min aur max dono hai, jo dikhata hai ki FONC poori continuum se satisfy ho sakta hai, sirf ek akele point se nahi.

Gradient

x^{*}

par exist karta hai lekin discontinuous hai, phir bhi wahan

0

ke barabar hai. Kya

x^{*}

phir bhi valid FONC candidate hai?

Haan — FONC sirf

x^{*}

par differentiability aur

\nabla f (x^{*}) = 0

maangta hai; nearby

\nabla f

ki continuity necessary condition ka hissa nahi hai.

f (x) = x^{4}

x = 0

par: gradient zero, lekin Hessian (yahan

f^{''} (0) = 0

) bhi zero hai. Tum kya conclude kar sakte ho?

Second-order test inconclusive hota hai jab Hessian sirf semi-definite/zero ho (Figure 3 ka right column); yahan higher-order analysis dikhata hai ki

0

genuinely minimum hai, lekin Hessian akele certify nahi kar sakta.

Ek stationary point jahan Hessian positive semidefinite hai (ek eigenvalue exactly

0

). Min guaranteed hai?

Nahi — positive semidefinite matlab curvature upward ya flat hai lekin kabhi downward nahi; yeh min ya degenerate saddle (jaise

f = x^{2} + y^{3}

) dono ke saath consistent hai, toh decide karne ke liye higher-order terms chahiye (Figure 3 ka inconclusive column).

Do variables,

\nabla f = 0

, Hessian eigenvalues dono negative. Point kya hai?

Strict local maximum — negative definite matlab

f

har direction mein downward curve karti hai (Figure 3 ka top-left cell), toh nearby saari values

f (x^{*})

se chhoti hain.

Koi bhi stationary point nahi rakhne wala function, jaise

f (x) = x

R

par. Kya iska koi interior optimum hai?

Nahi — kyunki

\nabla f

kabhi zero nahi hota, FONC ki necessary condition kabhi meet nahi hoti, toh koi interior max ya min nahi hai (values

\pm \infty

ki taraf bhaag jaati hain).

Recall Ek-line survival summary

FONC ek filter hai, verdict nahi: $\nabla f = 0$ flat points (mins, maxes, saddles, inflections) collect karta hai, Hessian unhe sort karta hai (Figure 3), boundaries ko Lagrange multipliers chahiye (Figure 4), aur sirf convexity ek stationary point ko global champion banati hai.

Connections

First-order optimality conditions — gradient = 0 — woh parent jise yeh bank drill karta hai.
Gradient and directional derivatives — har "why" ke peeche ka slope tool.
Hessian matrix and second-order conditions — woh traps resolve karta hai jo FONC chhod jaata hai.
Saddle points — classic false positive.
Convex functions — jahan stationary really matlab global hota hai.
Lagrange multipliers — boundary escape hatch.