First-order optimality conditions — gradient = 0: Worked examples

4.10.18 · D3 · Maths › Advanced Topics (Elite Level) › First-order optimality conditions — gradient = 0

Prerequisites jo tumhe pehle se pata hone chahiye: Gradient and directional derivatives ( $\nabla f$ kya hota hai), Hessian matrix and second-order conditions (second derivatives ka $2 \times 2$ matrix jo point classify karta hai), aur Saddle points. Baaki sab yahan build kiya gaya hai.

Definition Eigenvalue — recap, kyunki 2-D tests isi par depend karte hain

Hessian $H$ woh $2 \times 2$ table hai jisme second derivatives hote hain $(f_{xx} f_{x y} f_{x y} f_{y y})$ ; har entry ek curvature hai. $H$ ka eigenvalue ek aisa number $λ$ hai jis par surface ko ek special direction mein squeeze karne par curvature exactly $λ$ milti hai — koi twisting nahi, pure bending. Stationary point ke paas surface ka picture karo: do perpendicular "principal" directions hote hain, aur eigenvalues batate hain ki surface kitni sharply upar ( $λ > 0$ ) ya neeche ( $λ < 0$ ) bend karti hai un directions mein. Toh:

dono $λ > 0$ → har direction mein upar bend → minimum (bowl);
dono $λ < 0$ → har jagah neeche bend → maximum (dome);
opposite signs → ek taraf upar, doosri taraf neeche → saddle;
ek $λ = 0$ → ek direction mein flat → degenerate, test khamosh hai.

Inhe padhna do friendly cases mein aasaan hai: agar $H$ diagonal hai, toh eigenvalues sirf dono diagonal entries hain; zyada generally ye $det (H - λ I) = 0$ solve karke milte hain, aur ek quick shortcut hai ki $det H = λ_{1} λ_{2}$ — toh $det H < 0$ force karta hai opposite signs (ek saddle), koi computation ki zaroorat nahi.

Scenario matrix

Intuition Is table ko kaise padhna hai (ek row picture ke saath walk karo)

Har row ek tarah ka stationary point hai jo tum encounter kar sakte ho — goal ye hai ki is page ke baad koi bhi cell tumhe surprise na kare. Column 3 ("Signal") tumhara decision rule hai: ye ek single fact (ek sign, ek determinant, ek eigenvalue) naam leta hai jo us case ko settle karta hai. Column 4 fully worked example ki taraf point karta hai. Abhi row D par practice karo, figure dekhte hue: case class hai "2-D minimum", signal hai "both eigenvalues $> 0$ ", aur shape figure ke top-left mein upward bowl hai. Trace karo: $\nabla f = 0$ solve karo dot par land karne ke liye, dono Hessian eigenvalues padho, dekho ki dono positive hain, aur conclude karo "bowl → minimum". Baaki har row same ritual hai alag signal ke saath.

Cell	Case class	Stationary point par Signal	Example
A	Clean minimum (1-D)	$f^{''} > 0$	Ex 1
B	Clean maximum (1-D)	$f^{''} < 0$	Ex 1 (same function)
C	Degenerate 1-D ( $f^{''} = 0$ ) — inflection vs flat min	$f^{''} = 0$ , higher probe zaroori	Ex 2
D	2-D minimum (positive-definite Hessian)	dono eigenvalues $> 0$	Ex 3
E	2-D maximum (negative-definite Hessian)	dono eigenvalues $< 0$	Ex 4
F	2-D saddle (indefinite Hessian)	eigenvalues opposite sign	Ex 5
G	2-D degenerate (singular Hessian) — poori valley line	ek eigenvalue $= 0$	Ex 6
H	Multiple stationary points, mixed types	har ek ko alag classify karo	Ex 7
I	Boundary / constrained — gradient ka vanish karna zaroori nahi	edge check karo, sirf interior nahi	Ex 8
J	Word problem (real quantity optimise karna hai)	model banao phir FONC apply karo	Ex 9

Chaar 2-D shapes (cells D–G)

Chaar 2-D outcomes exactly chaar surface shapes hain neeche. Figure ko $2 \times 2$ grid ki tarah padho, aur har panel ko uske cell se map karo:

Top-left, blue bowl → cell D (minimum): dono eigenvalues $> 0$ , har slice upar curve karti hai; black dot lowest point par baitha hai.
Top-right, orange dome → cell E (maximum): dono eigenvalues $< 0$ , har slice neeche curve karti hai; dot summit par hai.
Bottom-left, red saddle → cell F (saddle): opposite sign ke eigenvalues; surface ek ridge ke along upar uthti hai aur perpendicular ridge ke along girती hai, dot par milti hain.
Bottom-right, green valley → cell G (degenerate): ek eigenvalue $0$ hai, isliye trough poori line ke along flat hai — dot infinitely many lowest points mein se sirf ek hai.

Har panel mein black dot wahan hai jahan $\nabla f = 0$ hai; uske aas-paas ki shape wohi hai jo eigenvalue signs secretly describe kar rahe hain. Ex 3–6 karte waqt ye picture saamne rakhna.

Figure — First-order optimality conditions — gradient = 0

Cells A & B — sabse clean 1-D case

Worked example

$f (x) = \frac{1}{3} x^{3} - 4 x$ ke har stationary point ko classify karo Forecast: pehle guess karo — kitne stationary points hain, aur kya har ek max hai, min hai, ya kuch nahi?

Step 1. Differentiate karo: $f^{'} (x) = x^{2} - 4$ . Ye step kyun? FONC kehta hai candidates wahan hain jahan derivative zero hai, isliye hume pehle $f^{'}$ chahiye.

Step 2. $f^{'} (x) = 0$ solve karo: $x^{2} = 4 \Rightarrow x = \pm 2$ . Kyun? Ye wahi interior points hain jahan "koi bhi direction downhill nahi" hold ho sakta hai.

Step 3. Second derivative $f^{''} (x) = 2 x$ . Kyun? FONC max ko min se alag nahi bata sakta (dono flat hain). $f^{''}$ ka sign measure karta hai ki tangent upar curve karti hai (bowl) ya neeche (dome).

Step 4. Evaluate karo:

$f^{''} (2) = 4 > 0$ → upar curve karta hai → local minimum (cell A). Value $f (2) = \frac{8}{3} - 8 = - \frac{16}{3}$ .
$f^{''} (- 2) = - 4 < 0$ → neeche curve karta hai → local maximum (cell B). Value $f (- 2) = - \frac{8}{3} + 8 = \frac{16}{3}$ .

Verify: $x = 2$ ke thoda left, jaise $x = 1.9$ , $f^{'} (1.9) = 3.61 - 4 = - 0.39 < 0$ (neeche ja raha hai); thoda right, $x = 2.1$ , $f^{'} (2.1) = 4.41 - 4 = 0.41 > 0$ (upar ja raha hai). Neeche-phir-upar = valley = minimum. ✓ $x = - 2$ par sign flip opposite hai. ✓

Cell C — degenerate 1-D trap ( $f^{''} = 0$ )

Worked example (i)

$f (x) = x^{3}$ aur (ii) $g (x) = x^{4}$ ke liye $x = 0$ classify karo Forecast: dono $0$ par flat hain zero curvature ke saath — kya dono same behave karte hain?

Step 1. $f^{'} (x) = 3 x^{2} \Rightarrow f^{'} (0) = 0$ ; $g^{'} (x) = 4 x^{3} \Rightarrow g^{'} (0) = 0$ . Dono stationary hain. Kyun? Confirm karo ki hum FONC shortlist mein hain hi.

Step 2. $f^{''} (x) = 6 x \Rightarrow f^{''} (0) = 0$ ; $g^{''} (x) = 12 x^{2} \Rightarrow g^{''} (0) = 0$ . Dono second-order tests khamosh hain. Ye step kyun? Ye exactly degenerate cell hai — hum yahan ruk nahi sakte.

Step 3. Higher derivatives probe karo. Pehla non-zero:

$f$ : $f^{'''} (0) = 6 \neq = 0$ . Odd-order derivative pehla non-zero hona ⟹ inflection (na max na min).
$g$ : $g^{''''} (0) = 24 > 0$ . Even order, positive ⟹ minimum. Kyun? Sabse chhota surviving term local shape govern karta hai: $f \approx x^{3}$ (odd, sign change karta hai) vs $g \approx x^{4}$ (even, hamesha $\geq 0$ ).

Verify: $f (- 0.1) = - 0.001 < 0 < 0.001 = f (0.1)$ — sign flip hoti hai, toh extremum nahi ✓. $g (\pm 0.1) = 0.0001 > 0 = g (0)$ — dono sides zyada hain, toh minimum ✓.

Cell D — 2-D bowl (positive-definite Hessian)

Worked example

$f (x, y) = x^{2} + 3 y^{2} - 2 x + 6 y + 10$ minimise karo Forecast: squares ka sum disguise mein — ek stationary point, aur min hona chahiye. Kahan?

Step 1. Gradient: $\nabla f = (2 x - 2 6 y + 6)$ . Kyun? Har entry $\partial f / \partial x_{i}$ hai; FONC ko poora vector vanish chahiye.

Step 2. $\nabla f = 0$ set karo: $2 x - 2 = 0 \Rightarrow x = 1$ ; $6 y + 6 = 0 \Rightarrow y = - 1$ . Kyun? Unique interior candidate $(1, - 1)$ hai.

Step 3. Hessian $H = (2006)$ . Ye tool kyun? 2-D mein ek number ( $f^{''}$ ) ka sign ek matrix of curvatures ban jaata hai; uski definiteness shape batati hai.

Step 4. Diagonal Hessian, toh eigenvalues sirf diagonal entries $2$ aur $6$ hain, dono $> 0$ ⟹ positive definite ⟹ har direction mein upar curve ⟹ local (yahan global) minimum (cell D). Ye upar figure ke top-left mein upward bowl hai. Value $f (1, - 1) = 1 + 3 - 2 - 6 + 10 = 6$ .

Verify (complete the square): $f = (x - 1)^{2} + 3 (y + 1)^{2} + 6$ , squares ka sum $\geq 6$ , sirf $(1, - 1)$ par $6$ ke barabar. ✓ Match karta hai.

Cell E — 2-D dome (negative-definite Hessian)

Worked example

$f (x, y) = - x^{2} - 2 y^{2} + 4 x + 4 y + 1$ ka stationary point dhundo aur classify karo Forecast: har squared term negative hai — downward dome. Single maximum expect karo.

Step 1. Gradient: $\nabla f = (- 2 x + 4 - 4 y + 4)$ . Kyun? FONC ko poora gradient pehle chahiye.

Step 2. $\nabla f = 0$ set karo: $- 2 x + 4 = 0 \Rightarrow x = 2$ ; $- 4 y + 4 = 0 \Rightarrow y = 1$ . Unique candidate $(2, 1)$ . Kyun? Linear equations, ek solution.

Step 3. Hessian $H = (- 2 0 0 - 4)$ . Ye tool kyun? " $f^{''} < 0$ ⟹ maximum" ka 2-D analogue. Ab hume poori matrix negative definite honi chahiye.

Step 4. Diagonal, toh eigenvalues $- 2$ aur $- 4$ hain, dono $< 0$ ⟹ negative definite ⟹ har direction mein neeche curve ⟹ local maximum (cell E). Ye upar figure ke top-right mein downward dome hai. Value $f (2, 1) = - 4 - 2 + 8 + 4 + 1 = 7$ . Cell D se test alag kyun hai: dono negative eigenvalues matlab point se har direction downhill hai — bowl ka exact mirror image.

Verify (complete the square): $f = - (x - 2)^{2} - 2 (y - 1)^{2} + 7$ . Dono negative squares matlab $f \leq 7$ , equality sirf $(2, 1)$ par. ✓ Genuine maximum of value $7$ .

Cell F — 2-D saddle (indefinite Hessian)

Worked example

$f (x, y) = x^{2} - 4 x y + y^{2}$ ke stationary point ko classify karo Forecast: cross term $- 4 x y$ bada hai — saddle ka smell aa raha hai.

Step 1. $\nabla f = (2 x - 4 y - 4 x + 2 y) = 0$ . Kyun? FONC.

Step 2. Solve karo: $2 x - 4 y = 0 \Rightarrow x = 2 y$ ; substitute karo $- 4 (2 y) + 2 y = - 6 y = 0 \Rightarrow y = 0$ , toh $x = 0$ . Sirf ek stationary point: $(0, 0)$ . Kyun? Linear system, unique solution.

Step 3. Hessian $H = (2 - 4 - 4 2)$ . Uska determinant $det H = 2 \cdot 2 - (- 4) (- 4) = 4 - 16 = - 12 < 0$ . Ye tool kyun? $2 \times 2$ Hessian ka negative determinant force karta hai opposite sign ke eigenvalues (recall $det H = λ_{1} λ_{2}$ ) — definitive saddle signature.

Step 4. Eigenvalues solve karo $(2 - λ)^{2} - 16 = 0 \Rightarrow λ = 2 \pm 4 = 6, - 2$ . Opposite signs ⟹ saddle (cell F): ek axis ke along upar, doosre ke along neeche. Ye upar figure ke bottom-left mein saddle shape hai.

Verify: line $y = x$ ke along: $f = x^{2} - 4 x^{2} + x^{2} = - 2 x^{2} < 0$ (girta hai). $y = - x$ ke along: $f = x^{2} + 4 x^{2} + x^{2} = 6 x^{2} > 0$ (uthta hai). Dono behaviours through $(0, 0)$ = saddle. ✓ Eigenvalues $6, - 2$ in dono directions se match karte hain.

Cell G — degenerate 2-D valley (singular Hessian)

Worked example

$f (x, y) = (x - y)^{2}$ ke stationary points classify karo Forecast: ye ek square hai, toh $f \geq 0$ — lekin exactly zero kahan hai, aur kitne "minima" hain?

Step 1. $\nabla f = (2 (x - y) - 2 (x - y)) = 0$ . Kyun? FONC.

Step 2. Dono equations reduce hoke same condition $x - y = 0$ deti hain, yaani $x = y$ . Toh stationary set poori line $y = x$ hai, single point nahi (cell G). Ye kyun matter karta hai: FONC tumhe candidates ka curve de sakta hai, sirf isolated dots nahi.

Step 3. Hessian $H = (2 - 2 - 2 2)$ , $det H = 4 - 4 = 0$ ⟹ singular ⟹ ek eigenvalue $0$ hai ⟹ Hessian test inconclusive hai. Kyun? Zero determinant matlab ek direction mein koi curvature nahi — flat trough — isliye second-order matrix akela kaam khatam nahi kar sakta. Ye upar figure ke bottom-right mein flat-bottomed valley hai.

Step 4. Directly argue karo: $f = (x - y)^{2} \geq 0$ har jagah, aur $y = x$ par exactly $0$ equals karta hai. Toh us line ka har point ek (non-strict, global) minimum hai — flat-bottomed valley (cell G).

Verify: $H$ ke eigenvalues $4$ aur $0$ hain (trace $4$ , det $0$ ). $0$ -eigenvalue direction $(1, 1)$ ke along hai — sach mein $y = x$ ke along move karne par $f = 0$ constant rehta hai. ✓

Cell H — multiple stationary points of mixed type

Worked example

$f (x, y) = x^{3} - 3 x + y^{2}$ ke SAARE stationary points dhundo aur classify karo Forecast: $x$ mein cubic do $x$ -candidates hint karta hai; $y$ mein parabola simple min hai. Ek min aur ek saddle expect karo.

Step 1. $\nabla f = (3 x^{2} - 3 2 y)$ . Kyun? FONC ko poora gradient chahiye.

Step 2. $= 0$ set karo: $2 y = 0 \Rightarrow y = 0$ ; $3 x^{2} - 3 = 0 \Rightarrow x = \pm 1$ . Do stationary points: $(1, 0)$ aur $(- 1, 0)$ . Kyun? Component-wise solve karo; $y$ -equation easy hai, $x$ -equation do roots deta hai.

Step 3. Hessian $H = (6 x 0 02)$ . Kyun? Har point alag classify karo — entry $6 x$ unke beech sign change karta hai.

Step 4. Evaluate karo (diagonal ⟹ eigenvalues diagonal entries hain):

$(1, 0)$ par: $H = (6002)$ , eigenvalues $6, 2 > 0$ ⟹ local minimum (bowl, cell D-type). Value $f (1, 0) = 1 - 3 + 0 = - 2$ .
$(- 1, 0)$ par: $H = (- 6 0 02)$ , eigenvalues $- 6, 2$ opposite signs ⟹ saddle (cell F-type). Value $f (- 1, 0) = - 1 + 3 + 0 = 2$ .

Verify: $(1, 0)$ ke paas, $f (1 + h, 0) = (1 + h)^{3} - 3 (1 + h) = - 2 + 3 h^{2} + h^{3} \approx - 2 + 3 h^{2} > - 2$ (uthta hai) ✓ min. $(- 1, 0)$ ke paas $x$ ke along: $f (- 1 + h, 0) = 2 - 3 h^{2} + \dots < 2$ (girta hai); $y$ ke along: $f (- 1, k) = 2 + k^{2} > 2$ (uthta hai) — upar aur neeche = saddle ✓.

Cell I — boundary case (optimum par gradient vanish nahi karta)

Worked example Closed disk

${x^{2} + y^{2} \leq 4}$ par, $f (x, y) = x^{2} + y^{2}$ ka min aur max dhundo Forecast: origin se sabse door ke points max jeettenge — kya $\nabla f = 0$ unhe dhund payega?

Step 1. Interior candidates: $\nabla f = (2 x, 2 y) = 0 \Rightarrow (0, 0)$ , $f = 0$ deta hai — minimum, aur clearly maximum nahi. Kyun? FONC sirf interior flat points shortlist karta hai.

Step 2. Maximum ke liye, boundary circle $x^{2} + y^{2} = 4$ check karo. Wahan $f = x^{2} + y^{2} = 4$ edge par har jagah hai. Ye step kyun? FONC ki derivation boundary par apply nahi hoti, isliye hum directly $f$ evaluate karte hain.

Step 3. Compare karo: interior mein $f = 0$ ; boundary mein $f = 4$ . Maximum value $= 4$ , poori circle par attain hoti hai — un points par jahan $\nabla f = (2 x, 2 y)$ ki magnitude $(2 x)^{2} + (2 y)^{2} = 2 \cdot 2 = 4 \neq = 0$ hai (radially bahar point karta hai). Ye kyun matter karta hai: optimum ka non-zero gradient hai — FONC akela ise bilkul miss kar jaata (cell I).

Verify: $(2, 0)$ par, $f = 4$ aur $\nabla f = (4, 0) \neq = 0$ , seedha disk ke bahar point karta hai — aage jaana chahte ho lekin boundary rok leti hai. ✓

Cell J — real-world word problem

Worked example Ek rectangular open-top box (lid nahi) mein volume

$32 cm^{3}$ hona chahiye. Uska surface area minimise karo. Forecast: guess karo ki base square hai ya nahi aur box tall hai ya squat.

Step 1 (model). Base $x \times y$ , height $z$ lo, sab cm mein. Volume: $x y z = 32 \Rightarrow z = \frac{32}{x y}$ . Kyun? Do free variables mein reduce karo taaki 2-D FONC apply kar sakein.

Step 2 (objective). Open top surface = base + 4 walls: $S (x, y) = x y + 2 x z + 2 y z = x y + 2 x \cdot \frac{32}{x y} + 2 y \cdot \frac{32}{x y} = x y + \frac{64}{y} + \frac{64}{x} .$ Kyun? Constraint substitute karo taaki $S$ sirf $x, y$ par depend kare — ab ye unconstrained hai aur FONC apply hota hai.

Step 3 (FONC). Gradient zero karo: $\frac{\partial S}{\partial x} = y - \frac{64}{x ^{2}} = 0, \frac{\partial S}{\partial y} = x - \frac{64}{y ^{2}} = 0.$ Ye step kyun? Interior minimum par surface dono directions mein flat hoti hai — ye exactly $\nabla S = 0$ hai.

Step 4 (solve). Pehle equation se, $y = \frac{64}{x ^{2}}$ . Doosre mein substitute karo: $x = \frac{64}{y ^{2}} = \frac{64}{( 64/ x ^{2} ) ^{2}} = 64 \cdot \frac{x ^{4}}{6 4 ^{2}} = \frac{x ^{4}}{64} \Rightarrow x^{3} = 64 \Rightarrow x = 4,$ phir $y = 64/16 = 4$ aur $z = 32/ (4 \cdot 4) = 2$ . Kyun? Hum sirf physically sensible root $x, y > 0$ rakhte hain; ye ek candidate box deta hai.

Step 5 (classify). $(4, 4)$ par second derivatives: $S_{xx} = \frac{128}{x ^{3}} = 2$ , $S_{y y} = \frac{128}{y ^{3}} = 2$ , aur $S_{x y} = 1$ . Toh $H = (2112), det H = 2 \cdot 2 - 1 \cdot 1 = 3 > 0, S_{xx} = 2 > 0.$ Ye step kyun? FONC sirf shortlist karta hai; positive determinant aur positive top-left entry matlab dono eigenvalues positive hain ( $det H = λ_{1} λ_{2} > 0$ aur unka sum $= 4 > 0$ dono ko force karta hai $> 0$ ) ⟹ positive definite ⟹ genuine minimum, saddle nahi.

Verify (units + value): $x = y = 4 cm, z = 2 cm$ ; volume $= 4 \cdot 4 \cdot 2 = 32 cm^{3}$ ✓ (constraint se match karta hai). Surface $S = 16 + 16 + 16 = 48 cm^{2}$ . Note karo base square hai aur box itna hi aadha tall hai jitna wide — classic open-box answer. ✓

Flashcards

Jab

f^{''} (x^{*}) = 0

ho, tum kya karte ho?

Second-order test khamosh hai; pehla non-zero higher derivative probe karo (odd order ⇒ inflection, even & positive ⇒ min).

Hessian ka eigenvalue intuitively kya hota hai?

Surface ki pure curvature uske do principal directions mein se ek ke along; uska sign batata hai bends-up (+) ya bends-down (−).

2-D Hessian test jab

det H > 0

aur

f_{xx} > 0

Positive definite ⇒ local minimum.

2-D Hessian test jab

det H > 0

aur

f_{xx} < 0

Negative definite ⇒ local maximum.

2-D Hessian test jab

det H < 0

Indefinite ⇒ saddle (opposite sign ke eigenvalues, kyunki

det H = λ_{1} λ_{2} < 0

2-D Hessian test jab

det H = 0

Inconclusive (singular) — flat direction; doosre means se classify karo.

Optimum kahan chhup sakta hai jab bhi

\nabla f \neq = 0

wahan ho?

Closed domain ki boundary par — FONC sirf interior points govern karta hai.

Volume

32 cm^{3}

wale open-top box ki minimum surface dimensions?

4 \times 4 \times 2

cm, surface

48 cm^{2}

Recall Poori matrix ki one-line summary

Dhundo jahan $\nabla f = 0$ ho (shayad zero, ek, kai, ya poori line of points), phir Hessian ke eigenvalue signs decide karein min/max/saddle — aur agar Hessian singular ho ya tum boundary par ho, toh haath se argue karo ya constraints use karo.

Connections

Parent: FONC — woh condition jo har example apply karta hai.
Hessian matrix and second-order conditions — cells C–J mein use kiya gaya classifier.
Saddle points — cell F concretely banaya gaya.
Convex functions — jab ek single stationary point (cell D) automatically global hota hai.
Lagrange multipliers — boundary cell I ka proper tool.
Gradient descent — ek algorithm jo minima mein settle ho jaata lekin saddles par stuck ho jaata.

Worked examples — First-order optimality conditions — gradient = 0

Scenario matrix

Chaar 2-D shapes (cells D–G)

Cells A & B — sabse clean 1-D case

Cell C — degenerate 1-D trap ( $f^{''} = 0$ )

Cell D — 2-D bowl (positive-definite Hessian)

Cell E — 2-D dome (negative-definite Hessian)

Cell F — 2-D saddle (indefinite Hessian)

Cell G — degenerate 2-D valley (singular Hessian)

Cell H — multiple stationary points of mixed type

Cell I — boundary case (optimum par gradient vanish nahi karta)

Cell J — real-world word problem

Flashcards

Connections

Concept Map