Gradient as direction of steepest ascent

---

WHAT are we even asking?

WHAT we want: out of all possible directions $\mathbf u$ you could walk, which one makes $f$ grow the fastest, and how fast is that?

---

HOW: derive the directional derivative from scratch

WHY this step? We don't want to take a limit for every direction. Use the multivariable chain rule to express $D_{\mathbf u}f$ using quantities we already know (the partials).

Define a single-variable function along the line through $\mathbf x$ in direction $\mathbf u$: $g(h) = f(\mathbf x + h\mathbf u), \qquad \mathbf u=(u_1,u_2)$

Then by definition $D_{\mathbf u}f = g'(0)$.

Apply the chain rule ($x = x_0+hu_1$, $y=y_0+hu_2$):

= f_x\,u_1 + f_y\,u_2$$ **WHY this step?** $\frac{dx}{dh}=u_1$ and $\frac{dy}{dh}=u_2$ because $x,y$ move linearly with $h$. Set $h=0$: $$\boxed{D_{\mathbf u}f = f_x u_1 + f_y u_2 = \nabla f \cdot \mathbf u}$$ > [!formula] Key result > $$D_{\mathbf u}f = \nabla f \cdot \mathbf u \quad (\|\mathbf u\|=1)$$ > The directional derivative is just the **dot product** of the gradient with the direction. --- ## HOW: now find the *steepest* direction We want to **maximise** $D_{\mathbf u}f = \nabla f \cdot \mathbf u$ over all unit $\mathbf u$. Use the geometric form of the dot product: $$\nabla f \cdot \mathbf u = \|\nabla f\|\,\|\mathbf u\|\cos\theta = \|\nabla f\|\cos\theta$$ **WHY this step?** $\|\mathbf u\|=1$, so only the angle $\theta$ between $\nabla f$ and $\mathbf u$ matters. Since $\cos\theta \in [-1,1]$: | $\theta$ | $\cos\theta$ | $D_{\mathbf u}f$ | meaning | |---|---|---|---| | $0$ (along $\nabla f$) | $+1$ | $+\|\nabla f\|$ | **steepest ascent** ↑ | | $\pi$ (opposite) | $-1$ | $-\|\nabla f\|$ | steepest descent ↓ | | $\pi/2$ (perpendicular) | $0$ | $0$ | along a level curve (flat) | > [!formula] The three big facts > - Steepest ascent direction: $\displaystyle \mathbf u^* = \frac{\nabla f}{\|\nabla f\|}$ > - Maximum rate of increase: $\displaystyle \max_{\mathbf u} D_{\mathbf u}f = \|\nabla f\|$ > - $\nabla f$ is **perpendicular** to level curves $f=\text{const}$ (because moving along a level curve gives $D_{\mathbf u}f=0 \Rightarrow \cos\theta=0$). --- ## Worked examples > [!example] Example 1 — Find the steepest direction > Let $f(x,y)=x^2+y^2$ at point $(1,2)$. > > **Step 1.** $f_x = 2x,\ f_y=2y$ → *Why?* take partials. > **Step 2.** $\nabla f(1,2) = (2,4)$ → *Why?* plug in the point. > **Step 3.** Steepest ascent direction $= \dfrac{(2,4)}{\sqrt{20}} = \dfrac{1}{\sqrt5}(1,2)$ → *Why?* normalise to unit length. > **Step 4.** Max rate $= \|\nabla f\| = \sqrt{4+16}=\sqrt{20}=2\sqrt5$ → *Why?* the maximum directional derivative equals the gradient's magnitude. > [!example] Example 2 — Directional derivative in a given direction > Same $f$, at $(1,2)$, direction towards $(4,6)$. > > **Step 1.** Direction vector $(4-1,\,6-2)=(3,4)$. *Why?* tip minus tail. > **Step 2.** Unit vector $\mathbf u = (3,4)/5$. *Why?* $\|\mathbf u\|$ must be 1, else the rate is scaled wrongly. > **Step 3.** $D_{\mathbf u}f = \nabla f\cdot\mathbf u = (2,4)\cdot(3/5,4/5)= \frac{6+16}{5}=\frac{22}{5}=4.4.$ *Why?* apply the boxed formula. > Note $4.4 < 2\sqrt5\approx4.47$ — consistent: nothing beats the gradient direction. > [!example] Example 3 — Perpendicular to level curve > $f(x,y)=xy$, point $(2,3)$. Level curve through it: $xy=6$. > > **Step 1.** $\nabla f=(y,x)=(3,2)$. *Why?* partials of $xy$. > **Step 2.** Tangent to $xy=6$: implicit diff $y + x\,y'=0 \Rightarrow y' = -3/2$, so tangent direction $\propto(2,-3)$. *Why?* slope $dy/dx$. > **Step 3.** Check $\nabla f\cdot(2,-3)=(3)(2)+(2)(-3)=0$. ✓ *Why?* gradient ⟂ level curve, exactly as predicted. --- ## Common mistakes > [!mistake] "I'll just use the raw direction vector, not a unit vector." > **Why it feels right:** $\nabla f\cdot\mathbf v$ looks like a directional derivative. > **Why it's wrong:** if $\|\mathbf v\|\ne 1$, you get $\|\mathbf v\|$ times too big — that's a rate *per unit of $\mathbf v$'s length*, not per unit distance. > **Fix:** always **normalise**: $\mathbf u = \mathbf v/\|\mathbf v\|$ before dotting. > [!mistake] "The gradient points *along* the level curve, like a contour line." > **Why it feels right:** you walk along trails on a map, so gradients must follow them. > **Why it's wrong:** along a level curve $f$ doesn't change, so the *fastest-increase* arrow must point **off** the curve — perpendicular to it. > **Fix:** remember $D_{\mathbf u}f=0$ along the contour $\Rightarrow \cos\theta=0 \Rightarrow \perp$. > [!mistake] "Steepest ascent and steepest descent are unrelated directions." > **Fix:** descent is just $-\nabla f$ (opposite arrow). Gradient descent (ML) literally steps in $-\nabla f$. --- > [!recall]- Feynman: explain to a 12-year-old > Imagine you're standing on a bumpy hill in thick fog. You can't see far, but you can feel the slope under your feet in every direction. The **gradient** is like an arrow that magically points straight uphill — the steepest way up. Turn fully around and you're pointing the steepest way **down**. Walk sideways at a right angle to the arrow and the ground stays flat (you're walking around the hill at the same height). The **longer** the arrow, the steeper the hill there. > [!mnemonic] Remember it > **"GUS"** — **G**radient = **U**phill, **S**teepest. And **"normalise before you dot."** > Magnitude $\|\nabla f\|$ = *how steep*; direction = *which way up*. --- ## Recall checkpoint > [!recall] Can you answer without looking? > 1. Write $D_{\mathbf u}f$ as a dot product. > 2. Why is the max value of $D_{\mathbf u}f$ equal to $\|\nabla f\|$? > 3. What angle does $\nabla f$ make with a level curve? --- ### #flashcards/maths What is the directional derivative $D_{\mathbf u}f$ in terms of the gradient? ::: $D_{\mathbf u}f = \nabla f \cdot \mathbf u$ for a **unit** vector $\mathbf u$. In which direction does $\nabla f$ point? ::: The direction of **steepest ascent** (fastest increase of $f$). What is the maximum rate of increase of $f$ at a point? ::: $\|\nabla f\|$, the magnitude of the gradient. What direction gives steepest descent? ::: $-\nabla f$ (opposite the gradient). Why must $\mathbf u$ be a unit vector in $D_{\mathbf u}f=\nabla f\cdot\mathbf u$? ::: So the rate is "per unit distance"; otherwise it's scaled by $\|\mathbf u\|$. What angle is the gradient to a level curve $f=\text{const}$? ::: $90°$ — it's perpendicular, because $D_{\mathbf u}f=0$ along the curve. Derive $D_{\mathbf u}f$: what trick is used? ::: Chain rule on $g(h)=f(\mathbf x+h\mathbf u)$, then $g'(0)=f_xu_1+f_yu_2$. Why is $\nabla f\cdot\mathbf u$ maximised along $\nabla f$? ::: $=\|\nabla f\|\cos\theta$, maximal when $\cos\theta=1$, i.e. $\theta=0$. --- ## Connections - [[Directional Derivative]] - [[Partial Derivatives]] - [[Multivariable Chain Rule]] - [[Dot Product and Angle]] - [[Level Curves and Contour Maps]] - [[Gradient Descent (Machine Learning)]] - [[Tangent Plane and Linearization]] ## 🖼️ Concept Map ```mermaid flowchart TD F[Scalar field f] -->|partial derivatives| GRAD[Gradient grad f] U[Unit direction u] -->|step along| DD[Directional derivative] DD -->|defined as| LIM[Limit of difference quotient] GRAD -->|chain rule gives| KEY[D_u f = grad f dot u] U --> KEY KEY -->|geometric dot product| ANG[grad f times cos theta] ANG -->|cos theta = 1| ASC[Steepest ascent along grad f] ANG -->|cos theta = -1| DESC[Steepest descent] ANG -->|cos theta = 0| LEVEL[Along level curve, flat] ASC -->|max rate equals| MAG[Magnitude of grad f] LEVEL -->|implies| PERP[grad f perpendicular to level curves] ``` ## 🔊 Hinglish (regional understanding) > [!intuition]- Hinglish mein samjho > Socho tum ek pahaadi (hill) pe khade ho aur fog ke kaaran kuch dikh nahi raha. Har direction mein zameen ka slope feel kar sakte ho. **Gradient** $\nabla f$ ek arrow hai jo seedha upar ki taraf — yaani jahan function sabse tezi se badhta hai — point karta hai. Iski **length** $\|\nabla f\|$ batati hai ki kitni tezi se badh raha hai. Opposite direction $-\nabla f$ steepest descent hai (machine learning ka gradient descent yahi use karta hai). > > Maths kaise aaya? Directional derivative $D_{\mathbf u}f$ matlab direction $\mathbf u$ mein chalne pe $f$ kis rate se change hota hai. Chain rule lagao to milta hai $D_{\mathbf u}f = \nabla f \cdot \mathbf u$ — bas ek simple dot product. Ab dot product ko likho $\|\nabla f\|\cos\theta$ ($\mathbf u$ unit hone ki wajah se). $\cos\theta$ max tab hota hai jab $\theta=0$, yaani jab tum gradient ki hi direction mein chalo. Isliye gradient = steepest ascent, aur max rate = $\|\nabla f\|$. > > Ek important baat: level curve (contour, jahan $f$ constant hai) ke along chalne pe $f$ change nahi hota, to $D_{\mathbf u}f=0$, matlab $\cos\theta=0$, matlab gradient level curve ke **perpendicular** hota hai. Yeh fact har exam mein kaam aata hai. > > Do common galtiyan yaad rakho: (1) Direction vector ko hamesha **unit vector** banao (normalise karo) warna rate galat scale ho jata hai. (2) Gradient contour line ke along nahi, uske perpendicular point karta hai. Mantra: **"GUS — Gradient Uphill Steepest, aur dot karne se pehle normalise karo."** ![[audio/4.4.10-Gradient-as-direction-of-steepest-ascent.mp3]]