Exercises — Gradients and directional derivatives

(a) Vector. The gradient has one component per input variable. Here there are two inputs $x,y$, so $\nabla f$ is a 2-component arrow, not a number. The number is what you get after dotting with a direction. (b) Steepest slope $=\Vert \nabla f\Vert$. Magnitude of a vector $(a,b)$ is $\sqrt{a^2+b^2}$: $\Vert \nabla f\Vert =\sqrt{3^2+(-4{)}^2}=\sqrt{9+16}=\sqrt{25}=5.$ This is the largest value $D_{\mathbf{u}}f$ can ever reach, because $\Vert \nabla f\Vert \cos \theta$ maxes out at $\cos \theta =1$. (c) Steepest-ascent unit direction $=\frac{\nabla f}{\Vert \nabla f\Vert }=\frac{(3,-4)}{5}=(0.6,-0.8).$ Dividing a vector by its own length gives a length-1 arrow pointing the same way.

(a) True. A contour (level set) is a curve where $f$ keeps the same value. Same value $\Rightarrow$ no change $\Rightarrow D_{\mathbf{u}}f=0$. Since $D_{\mathbf{u}}f=\nabla f\cdot \mathbf{u}$, that dot product is zero, so $\mathbf{u}\perp \nabla f$. (b) False. The gradient is perpendicular to the contour, not along it. It points across the contours, straight uphill. See the trap below.

Partials — differentiate one variable, freeze the other (see Partial derivatives). $\frac{\partial f}{\partial x}=2xy$ (treat $y$ constant; $y^3$ has no $x$, so it dies). $\frac{\partial f}{\partial y}=x^2+3y^2$ (treat $x$ constant). Evaluate at $(1,2)$: $\nabla f(1,2)=(2\cdot 1\cdot 2,1^2+3\cdot 2^2)=(4,13).$

Step 1 — raw direction (arrow from $\mathbf{a}$ to target): $(4-1,6-2)=(3,4)$. Step 2 — normalise. Length $=\sqrt{3^2+4^2}=5$, so the unit step is $\mathbf{u}=(3/5,4/5)=(0.6,0.8)$. Why normalise? $D_{\mathbf{u}}f$ should measure slope per one step of distance; a longer arrow would fake a steeper slope. See Dot product and projections. Step 3 — dot product with $\nabla f=(4,13)$ from 2.1: $D_{\mathbf{u}}f=(4,13)\cdot (0.6,0.8)=2.4+10.4=\mathrm{12.8.}$

Gradient (1-D, so it's an ordinary derivative): $L^{\prime }(w)=2(w-3)+\frac{1}{2}$. At $w=5$: $L^{\prime }(5)=2(2)+0.5=4.5$. Positive $\Rightarrow$ loss rises as $w$ grows, so we step the other way. Update: $w\leftarrow w-\eta L^{\prime }(w)=5-0.1(4.5)=5-0.45=\mathrm{4.55.}$ We moved toward smaller loss by stepping opposite the gradient, because $-\nabla L$ is steepest descent.

Use $D_{\mathbf{u}}f=\Vert \nabla f\Vert \cos \theta$ with $\Vert \nabla f\Vert =\sqrt{36+64}=\sqrt{100}=10$. (a) Max at $\cos \theta =1$ ($\theta =0$, step aligned with gradient): $\mathbf{u}=(6,8)/10=(0.6,0.8)$, value $=+10$. (b) Min at $\cos \theta =-1$ ($\theta =180^{\circ }$, step opposite): $\mathbf{u}=(-0.6,-0.8)$, value $=-10$. This is the descent direction. (c) Zero at $\cos \theta =0$ ($\theta =90^{\circ }$, step perpendicular): $\mathbf{u}=(0.8,-0.6)$ or $(-0.8,0.6)$, value $=0$ — both run along the contour.

Notice the two contour directions in (c): perpendicularity has two solutions (forward and backward along the contour), and both give zero.

Dotting the gradient with an axis unit vector just reads off a component: $\nabla f\cdot (1,0)=f_x=5$ and $\nabla f\cdot (0,1)=f_y=12$. So $\nabla f(\mathbf{a})=(5,12)$, and the steepest slope is $\Vert \nabla f\Vert =\sqrt{5^2+12^2}=\sqrt{25+144}=\sqrt{169}=13.$ The directional derivatives along the axes are literally the partial derivatives — that's why the gradient is the vector of partials.

Gradient: $f_x=2x+y$, $f_y=x+4y$, so $\nabla f(2,1)=(5,6)$, $\Vert \nabla f\Vert =\sqrt{25+36}=\sqrt{61}$. Half the max rate means $D_{\mathbf{u}}f=\frac{1}{2}\Vert \nabla f\Vert$. But $D_{\mathbf{u}}f=\Vert \nabla f\Vert \cos \theta$, so $\Vert \nabla f\Vert \cos \theta =\frac{1}{2}\Vert \nabla f\Vert \Rightarrow \cos \theta =\frac{1}{2}\Rightarrow \theta =60^{\circ }.$ The rate depends only on the angle, not on where you stand — a clean payoff of $\Vert \nabla f\Vert \cos \theta$. One valid direction: rotate the unit gradient $\hat{g}=(5,6)/\sqrt{61}$ by $60^{\circ }$. Using the 2-D rotation of $\hat{g}=(\cos \alpha ,\sin \alpha )$ by $60^{\circ }$: any $\mathbf{u}$ with $\mathbf{u}\cdot \hat{g}=\cos 60^{\circ }=0.5$ works. (A second valid answer exists by rotating $-60^{\circ }$.)

Parametrise a path $\mathbf{x}(t)$ that stays on the contour, and let $h(t)=f(\mathbf{x}(t))$ be the height along it. By the Chain rule, $h^{\prime }(t)={\sum }_i\frac{\partial f}{\partial x_i}\frac{d{x}_i}{dt}=\nabla f\cdot {\mathbf{x}}^{\prime }(t).$ On a contour the tangent ${\mathbf{x}}^{\prime }(t)$ is perpendicular to $\nabla f$ (that's what "along the contour" means), so the dot product is $0$. Hence $h^{\prime }(t)=0$ everywhere on the path $\Rightarrow h$ is constant $\Rightarrow f$ never changes. This is exactly the statement "$\nabla f\perp$ contour" turned into a derivative fact.

Gradient: $\nabla f=(2x,2y)$. At $(3,4)$: $\nabla f=(6,8)$, $\Vert \nabla f\Vert =10$, steepest-ascent unit direction $(0.6,0.8)$, max slope $10$. Contours are circles; the gradient points radially outward — perpendicular to the circle, as expected. At the origin $(0,0)$: $\nabla f=(0,0)$. Now $D_{\mathbf{u}}f=0\cdot \mathbf{u}=0$ for every direction $\mathbf{u}$. There is no steepest direction — the surface is momentarily flat to first order. This is a critical point (here, the minimum). The direction of steepest ascent is undefined; that's not a failure of the theory, it's the signal that gradient descent has converged (zero step). See Jacobian and Hessian for how the second-order info (the Hessian) tells minimum vs maximum vs saddle when the gradient is zero.

(a) Gradient: $\nabla L=(2w_1,8w_2)$. At $(-2,1)$: $\nabla L=(-4,8)$. (b) Step opposite the gradient: $\mathbf{w}\leftarrow \mathbf{w}-\eta \nabla L=(-2,1)-0.25(-4,8)=(-2+1,1-2)=(-1,-1).$ (c) Loss check: $L(-2,1)=4+4=8$; $L(-1,-1)=1+4=5$. Since $5<8$, the loss dropped. ✓ (d) Why not straight at $(0,0)$? The straight-to-minimum direction from $(-2,1)$ is $(2,-1)$ (normalised $(0.894,-0.447)$). The negative gradient is $-\nabla L=(4,-8)$ (normalised $(0.447,-0.894)$) — it leans much harder along $w_2$ because $L$ is 4× steeper in $w_2$ (the $4w_2^2$ term). Steepest descent chases steepness, not the shortest route home. On stretched (ill-conditioned) bowls this causes the zig-zag that motivates momentum and preconditioning.

L1 tested recognising gradient-as-vector and the contour-perpendicularity fact. L2 tested the compute pipeline: partials → evaluate → normalise → dot. L3 tested reasoning through $\Vert \nabla f\Vert \cos \theta$ (max/min/zero, two perpendicular solutions). L4 tested combining rate control via angle and proving the contour fact with the chain rule. L5 tested the degenerate zero-gradient case, a full sign audit, and minimum-vs-saddle nuance.