Calculus & Optimization Basics

Question 1

(a) [6 marks] Chain rule through the graph $L\leftarrow a\leftarrow z\leftarrow (w,b)$.

• $\dfrac{\partial L}{\partial a} = (a-y)$ [1]
• $\dfrac{\partial a}{\partial z} = 1-a^2$ (since $a=\tanh z$) [1]
• Upstream gradient: $\delta = \dfrac{\partial L}{\partial z} = (a-y)(1-a^2)$ [2]
• $\dfrac{\partial z}{\partial w}=x,\ \dfrac{\partial z}{\partial b}=1$, hence $\frac{\partial L}{\partial w}=\delta\, x=(a-y)(1-a^2)x,\qquad \frac{\partial L}{\partial b}=\delta=(a-y)(1-a^2).$ [2]

(b) [4 marks] Let $u=e^z-e^{-z}$, $v=e^z+e^{-z}$, so $\tanh z=u/v$. Note $u'=v$, $v'=u$. [1] Quotient rule: $\tanh'z=\dfrac{u'v-uv'}{v^2}=\dfrac{v^2-u^2}{v^2}=1-\dfrac{u^2}{v^2}=1-\tanh^2 z.$ [3] (Full marks for showing $v^2-u^2$ formation.)

(c) [6 marks]

• $z=wx+b=0.5\cdot 2+0=1$ [1]
• $a=\tanh(1)=0.76159$ [1]
• $a-y=0.76159$; $1-a^2=1-0.58002=0.41998$ [1]
• $\delta=(a-y)(1-a^2)=0.76159\cdot0.41998=0.31985$ [1]
• $\partial L/\partial w=\delta x=0.31985\cdot2=0.63970$ [1]
• $\partial L/\partial b=\delta=0.31985$ [1]

(d) [6 marks] Update ($\eta=1$): $w'=w-\eta\,\partial L/\partial w=0.5-0.63970=-0.13970$; $b'=0-0.31985=-0.31985$. [2] New $z'=-0.13970\cdot2-0.31985=-0.59925$, $a'=\tanh(-0.59925)=-0.5363$, new $L'=\tfrac12(a')^2=0.1438$; old $L=\tfrac12(0.76159)^2=0.2900$. Loss decreased. [2] Curvature/large-$\eta$ argument: $L$ as a function of $w$ has second derivative (local curvature) $\partial^2 L/\partial w^2$; gradient descent is stable when $\eta<2/\kappa$ where $\kappa$ is the local curvature. Too large $\eta$ overshoots the minimum, causing oscillation or divergence (the step jumps past the valley bottom and climbs the opposite wall). Full marks for linking overshoot to $\eta>2/\text{curvature}$. [2]

Question 2

(a) [7 marks] $\mathcal{L}=x^2+2y^2-\lambda(x+y-3)$. [1] Stationarity:

• $\partial_x: 2x-\lambda=0\Rightarrow x=\lambda/2$ [1]
• $\partial_y: 4y-\lambda=0\Rightarrow y=\lambda/4$ [1]
• constraint: $x+y=3$ [1] Sub: $\lambda/2+\lambda/4=3\Rightarrow 3\lambda/4=3\Rightarrow \lambda^*=4$. [1] So $x^*=2,\ y^*=1$. [2]

(b) [6 marks] $H_f=\begin{pmatrix}2&0\\0&4\end{pmatrix}$, positive definite (eigenvalues $2,4>0$). [3] Restricting to line $y=3-x$: $\phi(x)=x^2+2(3-x)^2=3x^2-12x+18$, $\phi''=6>0$ ⇒ minimum; $\phi'(x)=6x-12=0\Rightarrow x=2$ confirms $(2,1)$. [3]

(c) [4 marks] At optimum $\nabla f=\lambda\nabla g$: $\nabla f=(2x,4y)=(4,4)$; $\nabla g=(1,1)$; indeed $(4,4)=4(1,1)=\lambda^*\nabla g$. Gradients are parallel (level set of $f$ tangent to constraint line). [4]

(d) [3 marks] $H_f$ has eigenvalues $2,4>0$ ⇒ $f$ is (strictly) convex on $\mathbb{R}^2$. [2] A convex objective over an affine (hence convex) constraint set has a unique global minimum, so $(2,1)$ is the global constrained minimizer. [1]

Question 3

(a) [4 marks] $\nabla f=(3x^2-3y^2,\ -6xy)$. Set $=0$: $-6xy=0\Rightarrow x=0$ or $y=0$.

• If $x=0$: $-3y^2=0\Rightarrow y=0$.
• If $y=0$: $3x^2=0\Rightarrow x=0$. Unique critical point $(0,0)$. [4]

(b) [6 marks] $H=\begin{pmatrix}6x & -6y\\ -6y & -6x\end{pmatrix}$. At $(0,0)$: $H=\begin{pmatrix}0&0\\0&0\end{pmatrix}$. [3] Determinant $=0$, both eigenvalues $0$ ⇒ second-derivative test inconclusive. [2] The quadratic term vanishes; behaviour is governed by the cubic terms, giving a degenerate "monkey saddle" (three ascending, three descending sectors) — neither max, min, nor ordinary saddle. [1]

(c) [6 marks] At $(1,1)$: $f=1-3=-2$. $\nabla f(1,1)=(3-3,\,-6)=(0,-6)$. [2] $H(1,1)=\begin{pmatrix}6&-6\\-6&-6\end{pmatrix}$. Second-order Taylor with $\Delta=(x-1,y-1)$: $f\approx -2 + (0,-6)\!\cdot\!\Delta + \tfrac12\Delta^\top\!\begin{pmatrix}6&-6\\-6&-6\end{pmatrix}\!\Delta$ $= -2 -6(y-1) +3(x-1)^2 -6(x-1)(y-1) -3(y-1)^2.$ [2] Steepest ascent direction = $\nabla f/\|\nabla f\|=(0,-6)/6=(0,-1)$; magnitude of directional derivative $=\|\nabla f\|=6$. [2]

(d) [2 marks] Gradient is exactly zero, so the update $\theta\leftarrow\theta-\eta\nabla f$ leaves $\theta$ unchanged — GD stalls at the flat critical point. Escape mechanisms: stochastic-gradient noise (SGD), momentum, or a random perturbation / second-order (Newton/saddle-free) step. [2]

[
  {"claim":"Q1c: delta=(a-y)(1-a^2)=0.31985 for z=1,y=0","code":"z=Rational(1); a=tanh(z); delta=(a-0)*(1-a**2); result = abs(float(delta)-0.31985)<1e-4"},
  {"claim":"Q1c: dL/dw = delta*x = 0.63970 with x=2","code":"z=Rational(1); a=tanh(z); delta=a*(1-a**2); dLdw=delta*2; result = abs(float(dLdw)-0.63970)<1e-4"},
  {"claim":"Q1d: loss decreased after eta=1 update (L'<L)","code":"a=tanh(1); delta=a*(1-a**2); w=0.5-1.0*float(delta)*2; b=0-1.0*float(delta); import sympy as sp; zp=w*2+b; ap=sp.tanh(zp); Lp=0.5*ap**2; L=0.5*a**2; result = float(Lp)<float(L)"},
  {"claim":"Q2a: constrained minimizer x*=2,y*=1,lambda*=4","code":"x,y,lam=symbols('x y lam'); sol=solve([2*x-lam,4*y-lam,x+y-3],[x,y,lam],dict=True)[0]; result = sol[x]==2 and sol[y]==1 and sol[lam]==4"},
  {"claim":"Q2c: grad f = lambda* grad g at optimum: (4,4)=4*(1,1)","code":"gf=Matrix([2*2,4*1]); gg=Matrix([1,1]); result = gf==4*gg"},
  {"claim":"Q3a: unique critical point (0,0)","code":"x,y=symbols('x y'); f=x**3-3*x*y**2; sol=solve([diff(f,x),diff(f,y)],[x,y],dict=True); result = sol==[{x:0,y:0}]"},
  {"claim":"Q3b: Hessian at origin is zero matrix (test inconclusive)","code":"x,y=symbols('x y'); f=x**3-3*x*y**2; H=hessian(f,(x,y)).subs({x:0,y:0}); result = H==zeros(2,2)"},
  {"claim":"Q3c: grad f at (1,1) = (0,-6), norm 6","code":"x,y=symbols('x y'); f=x**3-3*x*y**2; g=Matrix([diff(f,x),diff(f,y)]).subs({x:1,y:1}); result = g==Matrix([0,-6]) and g.norm()==6"}
]