Visual walkthrough — First-order optimality conditions — gradient = 0

4.10.18 · D2 · Maths › Advanced Topics (Elite Level) › First-order optimality conditions — gradient = 0

---

Step 1 — "Slope" ka matlab aakhir hota kya hai?

KYA. Kisi bhi calculus se pehle, ek smooth hill ki curve ke roop mein imagine karo. Uska ek point chuno. Zoom in karo. Jaise-jaise aap zoom karte hain, curve zyada se zyada ek straight line jaisi dikhne lagti hai — uski tangent line. Us line ka slope ek single number hota hai: ek step right jaane par aap kitne steps upar jaate hain.

KYUN. Optimisation ka sawaal hai "kya main aur neeche ja sakta hoon?" — aur "kya main aur neeche ja sakta hoon" ka jawaab bilkul apne pairon ke neeche ke slope se milta hai. Agar zameen tilted hai, ek direction neeche jaata hai. Toh slope hi poora game hai; hume pehle ise pin down karna hoga.

PICTURE. Neeche figure mein, black curve hill hai. Magenta line marked point par tangent hai. Uska slope woh ratio hai (rise)/(run) jo orange triangle se dikhaya gaya hai.

---

Step 2 — Slope ko ek aise number mein badalna jo hum compute kar sakein

KYA. Point $x^{\ast }$ par slope measure karne ke liye, size $h$ ka ek tiny horizontal step lo aur dekho height kitni change hoti hai. Naye point par height hai $f(x^{\ast }+h)$; purane point par height hai $f(x^{\ast })$. Unka difference step ke upar woh difference quotient hai:

$$\underset{\text{rise over run}}{\underbrace{\frac{\stackrel{\text{height after step}}{\overbrace{f(x^{\ast }+h)}}-\stackrel{\text{height now}}{\overbrace{f(x^{\ast })}}}{\underset{\text{how far we stepped}}{\underbrace{h}}}}}$$

KYUN. Hum exact slope chahte hain, lekin "exact" ke liye step ko nothing mein shrink karna padta hai. Difference quotient real width $h$ ke step ke liye honest rise-over-run hai; $h\to 0$ shrink karna tangent ka slope deta hai — woh limit derivative $f^{\prime }(x^{\ast })$ hai, jo exist karta hai precisely isliye kyunki humne assume kiya tha ki $f$, $x^{\ast }$ par differentiable hai. Hum ek limit use karte hain (na ki $h=0$ plug karna) kyunki $h=0$ plug karne se $0/0$ milta hai, jo meaningless hai — limit woh tool hai jo exactly isi ko rescue karne ke liye bana hai.

PICTURE. Curve par do dots ek violet secant line se joined hain; jaise right dot left dot ki taraf slide karta hai, secant magenta tangent mein tip over ho jaati hai.

---

Step 3 — Setup: "local minimum" hume kya guarantee karta hai

KYA. Maano $x^{\ast }$ ek local minimum hai: koi bhi nearby point usse neeche nahi hai. Formally, ek small radius $\delta >0$ hai aise ki har step $h$ ke liye jahan $|h|<\delta$,

$$\underset{\text{any nearby height}}{\underbrace{f(x^{\ast }+h)}}\ge \underset{\text{our valley floor}}{\underbrace{f(x^{\ast })}}.$$

$|h|<\delta$ ka matlab hai "$h$ size mein $\delta$ se chhota hai, chahe $h$ positive ho ya negative" — right aur left dono taraf steps allowed hain.

KYUN. Yeh single inequality hi wo ek fact hai jo hum minimum ke baare mein jaante hain. Neeche sab kuch isi se squeeze kiya gaya hai. Notice karo woh crucial word interior: kyunki $x^{\ast }$ domain ke andar baitha hai, dono $+h$ aur $-h$ legal rehte hain. Yeh baat yaad rakhna — Step 6 mein yahi kaam aayega.

PICTURE. Ek flat floor wali valley; $x^{\ast }$ ke around $2\delta$ width ka ek shaded band; us band mein curve ka har point dashed floor line ke at or above baitha hai.

---

Step 4 — RIGHT taraf poke karo (ek one-sided limit, $h\to 0^{+}$)

KYA — pehle, $h\to 0^{+}$ ka matlab kya hai? Right se ek one-sided limit, ${\lim }_{h\to 0^{+}}$ likha jaata hai, yeh poochha jaata hai: jaise $h$ $0$ ki taraf shrink hota hai lekin strictly positive rehta hai (sirf right side se approach karta hai, $h=0.1,0.01,0.001,\dots$), toh woh quantity kis value par settle hoti hai? Yeh limit ka ordinary idea hai, lekin $0$ ke ek side tak restricted. Hum sides mein split karte hain precisely isliye kyunki "right step" ($h>0$) aur "left step" ($h<0$) hume alag-alag cheezein batayenge.

Ab right taraf steps lo, toh $h>0$, aur difference quotient ko piece by piece padho:

$$\frac{\stackrel{\ge 0\text{(min: nearby}\ge \text{floor)}}{\overbrace{f(x^{\ast }+h)-f(x^{\ast })}}}{\underset{>0\text{(stepping right)}}{\underbrace{h}}}\ge 0.$$

Top $\ge 0$ hai kyunki minimum matlab har neighbour kam se kam utna hi upar hai. Bottom $>0$ hai kyunki hum right step gaye. Ek non-negative number divided by positive number non-negative hota hai. $h\to 0^{+}$ karne par:

$$f^{\prime }(x^{\ast })=\underset{h\to 0^{+}}{\lim }(\cdots )\ge 0.$$

(Differentiability hume is one-sided limit ko $f^{\prime }(x^{\ast })$ kehne deti hai — dono one-sided limits single derivative ke equal hote hain.)

KYUN. Numbers ki ek limit jo sab $\ge 0$ hain, woh achanak negative nahi ho sakti — woh $\ge 0$ par land karti hai. Toh right side se akele, slope non-negative hai: tangent right ki taraf dekhne par neeche nahi ja sakti.

PICTURE. Dots ka right-hand track $x^{\ast }$ ki taraf right se creep kar raha hai; secant slopes sab $\ge 0$ hain, orange mein dikhaye gaye hain.

---

Step 5 — LEFT taraf poke karo (mirror one-sided limit, $h\to 0^{-}$)

KYA. Left se one-sided limit, ${\lim }_{h\to 0^{-}}$, Step 4 ka mirror hai: $h$ $0$ ki taraf shrink hota hai lekin strictly negative rehta hai ($h=-0.1,-0.01,-0.001,\dots$), sirf left se approach karta hai. Left step lo, toh $h<0$:

$$\frac{\stackrel{\ge 0\text{(still}\ge \text{floor)}}{\overbrace{f(x^{\ast }+h)-f(x^{\ast })}}}{\underset{<0\text{(stepping left)}}{\underbrace{h}}}\le 0.$$

Top abhi bhi $\ge 0$ hai (minimum dono directions mein minimum hota hai). Lekin ab bottom negative hai. Ek non-negative number divided by negative number $\le 0$ hota hai. Toh

$$f^{\prime }(x^{\ast })=\underset{h\to 0^{-}}{\lim }(\cdots )\le 0.$$

KYUN. Sirf $h$ ka sign flip hua, jo poore quotient ka sign flip kar deta hai. Left se, wahi derivative non-positive hone par majboor ho jaata hai.

PICTURE. Left-side secants, Step 4 ka mirror image, sab violet mein downward ya flat tilt karte hue jaise left dot $x^{\ast }$ ki taraf slide karta hai.

---

Step 6 — Squeeze: do inequalities crush hokar equality ban jaati hain

KYA. Kyunki $f$, $x^{\ast }$ par differentiable hai, left-side aur right-side one-sided limits same number $f^{\prime }(x^{\ast })$ hain. Step 4 ne bola tha yeh $\ge 0$ hai; Step 5 ne bola tha yeh $\le 0$ hai. Jo number dono ho woh hilne ki jagah nahi:

$$f^{\prime }(x^{\ast })\ge 0\text{and}{f}^{\prime }(x^{\ast })\le 0⟹\boxed{f^{\prime }(x^{\ast })=0.}$$

KYUN. Isliye hume dono directions chahiye thi, aur isliye point interior hona chahiye. Ek boundary par aap sirf ek taraf poke kar sakte ho — tumhe sirf Step 4 ya sirf Step 5 milega, kabhi dono nahi, aur slope nonzero reh sakta hai (outward point karta hua). Yeh two-sided poke hi poora engine hai.

PICTURE. $f^{\prime }(x^{\ast })$ value ke liye ek number line: ek orange arrow right se ise $\ge 0$ pin karta hai, ek violet arrow left se ise $\le 0$ pin karta hai; woh $0$ par ek single magenta dot par collide karte hain.

---

Step 7 — Many variables tak jaana, ek axis at a time

KYA. Ab $f$ several inputs leta hai, e.g. $f(x,y)$ — ek landscape, curve nahi. Ek ko chhod kar har coordinate freeze karo. Sirf $x_i$-axis ke along slide karna ek 1-D slice nikalta hai — uski height $g(t)=f(x^{\ast }+t{e}_i)$ kahlo, jahan $e_i$ axis $i$ ke along unit step hai aur $t$ aap kitna slide kiye ho.

Agar $x^{\ast }$ poore landscape ka lowest point hai, toh yeh certainly us single slice ke along lowest point hai, toh $t=0$, $g$ ko minimise karta hai. Step 6 se, $g^{\prime }(0)=0$. Aur $g^{\prime }(0)$ exactly partial derivative $\partial f/\partial x_i$ hai — $f$ ka slope agar aap sirf $i$-th input wiggle karo.

KYUN. Ek many-variable minimum, us se guzarne wali har one-dimensional line ke along minimum hota hai, toh har slice hume ek flat-slope equation deti hai. Har axis ke liye yeh karo aur sab partials vanish ho jaate hain. Ab hum un partial slopes ko ek single arrow mein bundle karte hain, gradient $\nabla f$ — woh vector jiska $i$-th entry $\partial f/\partial x_i$ hai (dekho Gradient and directional derivatives). Tab "all partials zero" exactly "gradient arrow ki zero length" hai:

$$\underset{\text{slope along axis }1}{\underbrace{\frac{\partial f}{\partial x_1}(x^{\ast })}}=\cdots =\underset{\text{slope along axis }n}{\underbrace{\frac{\partial f}{\partial x_n}(x^{\ast })}}=0⟺\nabla f(x^{\ast })=0.$$

PICTURE. Ek bowl-shaped surface; do perpendicular slice-curves base se guzar rahi hain, dono neeche flat tangent dikhaa rahi hain — ek $x$ ke along, ek $y$ ke along.

---

Step 8 — Sabse deep reason: koi bhi downhill direction survive nahi karta

KYA — pehle, do tools jo hume define karne honge.

Tool A — vector ki length (norm). Gradient $\nabla f(x^{\ast })$ (Step 7 mein bana) ek direction aur length wala arrow hai. Uski length double bars se likhi jaati hai, $\Vert \cdot \Vert$, aur Pythagoras se uske components par compute ki jaati hai:

$$\Vert v\Vert =\sqrt{v_1^2+v_2^2+\cdots +v_n^2}\ge 0.$$

Do facts jo hum use karte hain: norm kabhi negative nahi hota, aur woh zero sirf zero vector ke liye hota hai (zero-length arrow woh arrow hai jo kahin nahi point karta). Toh "$\Vert \nabla f\Vert >0$" sirf shorthand hai "$\nabla f$ ek genuine, nonzero arrow hai."

Tool B — directional derivative. Gradient steepest increase ki direction mein point karta hai. Directional derivative $D_{u}f(x^{\ast })$ jawaab deta hai: agar main ek chosen unit direction $u$ mein chalun (length $1$ ka arrow), toh $f$ kitni fast change hoti hai? Formula ek dot product hai:

$$D_{u}f(x^{\ast })=\underset{\text{steepest-up arrow}}{\underbrace{\nabla f(x^{\ast })}}\cdot \underset{\text{unit walking direction}}{\underbrace{u}}.$$

Dot product $\cdot$ dono vectors ki lengths ko unke beech ke angle ki cosine se multiply karta hai: yeh largest hota hai jab $u$ gradient ke saath point kare, zero jab $u$ perpendicular ho, aur most negative jab $u$ gradient ke against point kare.

Ab argument. Maan lo, contradiction ke liye, $\nabla f(x^{\ast })\ne 0$, yaani $\Vert \nabla f(x^{\ast })\Vert >0$. Deliberately seedha downhill chalo, unit direction choose karo

$$u=-\frac{\nabla f(x^{\ast })}{\Vert \nabla f(x^{\ast })\Vert }(\text{gradient reversed, shrunk to length }1).$$

Formula mein plug karo — ek vector ko apni scaled copy ke saath dot karne se uski length squared uski length se divide hoti hai:

$$D_{u}f(x^{\ast })=\nabla f\cdot \left(-\frac{\nabla f}{\Vert \nabla f\Vert }\right)=-\frac{\Vert \nabla f{\Vert }^2}{\Vert \nabla f\Vert }=-\Vert \nabla f(x^{\ast })\Vert <0.$$

Negative slope matlab hai $f$ decrease karti hai jaise aap chalte ho — toh $x^{\ast }$ minimum tha hi nahi. Contradiction. Ek hi escape: $\nabla f(x^{\ast })=0$.

KYUN. Yeh batata hai kyun zero gradient unavoidable hai: ek nonzero gradient literally ek arrow hai jis par lika hai "improvement is taraf rehta hai." Ek genuine optimum aisa arrow chhoda nahi reh sakta.

PICTURE. Ek non-optimal point par, magenta gradient arrow uphill point karta hai; orange $u=-\nabla f$ arrow surface ke along downhill point karta hai, ek aisa path trace karte hue jo visibly drop karta hai — prove karte hue ki woh point beatable tha.

---

Step 9 — Traps: flat aur bottom ek cheez nahi hain

KYA. Flatness ($\nabla f=0$) necessary hai, kabhi sufficient nahi. Teen alag shapes sab test pass karti hain:

• Minimum: flat, aur har direction mein rising (ek bowl).
• Maximum: flat, aur har direction mein falling (ek dome). Wahi proof — bas $\ge$ ko $\le$ se swap karo.
• Saddle: flat, phir bhi ek axis ke along rising aur doosre ke along falling. Example $f(x,y)=x^2-y^2$: $x$ ke along yeh climb karta hai, $y$ ke along drop karta hai, lekin $\nabla f(0,0)=0$ phir bhi. Dekho Saddle points.

1-D mein bhi, $f(x)=x^3$ mein $f^{\prime }(0)=0$ hai phir bhi $0$ ek inflection hai, na peak na valley.

KYUN. Humara proof sirf yeh rule out karta tha ki "ek direction jo cheezein immediately improve kare." Ek saddle mein hote hain woh directions jo improve karte hain — bas tested axis ke along flat point par right nahi. Isliye exactly Hessian second-order test exist karta hai: inhe teeno alag karne ke liye.

PICTURE. Teen mini-landscapes side by side — bowl, dome, saddle — har ek ke centre dot par flat magenta tangent plane, identical flatness dikhate hue lekin wildly different shapes ke saath.

---

Ek-picture summary

Sab ek saath: two-sided poke slope ko zero par trap karta hai (1-D), ise sab axes par stack karna $\nabla f=0$ deta hai (many-D), aur flat point sirf ek finalist hai — bowl, dome, ya saddle abhi decide hona baaki hai.

Recall Feynman: poora walkthrough plain words mein

Aap kisi smooth landscape par kahin khade ho aur jaanna chahte ho ki kya aap lowest spot par ho. Aap ek simple experiment karte ho: apna paon right nudge karo — agar aap min par ho, toh zameen drop nahi kar sakti, toh right-slope kam se kam zero hai ($\ge 0$). Left nudge karo — usi reasoning se zameen woh taraf bhi drop nahi kar sakti, lekin "left ki taraf drop nahi karna" matlab hai ki left ki taraf slope at most zero hai ($\le 0$). Yahan punchline hai: zameen ki ek slope hoti hai (yahi "smooth / differentiable" guarantee karta hai), aur woh ek number jo simultaneously "zero se neeche nahi" aur "zero se upar nahi" ho exactly zero hai. Toh zameen flat hai. Puri landscape mein aap bas yeh nudge har compass axis ke along repeat karo, aur har ek flat waapas aata hai — woh flat slopes ka bundle hai jo $\nabla f=0$ zero length ke single arrow mein pack karta hai. Ise dekhne ka sabse deep tarika: agar zameen flat nahi hoti, toh gradient ek real arrow hota jo chilla raha hota "downhill woh taraf hai," aur aap follow karte kahin aur lower — toh aap bottom par ho hi nahi sakte. Catch, last figure mein drawn: flat zameen ek hill ke top par aur ek horse-saddle ke middle mein bhi dikhti hai. Flat ek clue hai, verdict nahi. Verdict padhne ke liye aap jhukke curvature check karte ho — aur woh Hessian ka kaam hai.

---

Connections

• Parent: FONC (Hinglish) — woh result jo yeh page visually derive karta hai.
• Gradient and directional derivatives — Steps 7–8 isi par run karte hain.
• Hessian matrix and second-order conditions — Step 9 ke traps settle karta hai.
• Saddle points — middle trap explicitly banaya gaya.
• Lagrange multipliers — kya karna hai jab Step 6 ka "interior" boundary par fail ho jaaye.
• Gradient descent — literally Step 8 ke downhill arrow par chalta hai.
• Convex functions — woh ek duniya jahan flat actually global bottom guarantee karta hai.

Concept Map