Calculus & Optimization Basics

Question 1 (14 marks)

(a) Partial derivatives (3): $\nabla f = \big(3x^2 - 3y^2,\ -6xy + 4y^3\big).$

• $f_x = 3x^2-3y^2$ (1.5), $f_y = -6xy+4y^3$ (1.5).

(b) Critical points: set both to zero (4). From $f_x=0$: $x^2=y^2 \Rightarrow x=\pm y$. From $f_y=0$: $-6xy+4y^3=0 \Rightarrow 2y(-3x+2y^2)=0$, so $y=0$ or $2y^2=3x$.

• If $y=0$: then $x^2=0 \Rightarrow x=0$. Point $(0,0)$. (1)
• If $y\neq0$, $x=\pm y$ and $2y^2=3x$:
  • $x=y$: $2x^2=3x \Rightarrow x=3/2$, so $(3/2,3/2)$. (1.5)
  • $x=-y$: $2y^2=3x=-3y \Rightarrow 2y^2+3y=0\Rightarrow y=-3/2$ (y≠0), $x=3/2$, so $(3/2,-3/2)$. (1.5)

Critical points: $(0,0),\ (3/2,3/2),\ (3/2,-3/2)$.

(c) Hessian (4): $H=\begin{pmatrix} 6x & -6y \\ -6y & -6x+12y^2\end{pmatrix}.$ (2 for correct entries.) At origin $H=\begin{pmatrix}0&0\\0&0\end{pmatrix}$ — the Hessian is the zero matrix, so the second-derivative test is inconclusive (2). (Higher-order analysis needed; origin is a degenerate critical point.)

(d) GD update: $(x,y) \leftarrow (x,y) - \eta\nabla f$ (3). At $(1,0)$: $\nabla f = (3\cdot1 - 0,\ 0) = (3,0)$. New point $= (1 - 3\eta,\ 0)$.

Question 2 (10 marks)

(a) (5) $g_x = e^x\sin y$, $g_y=e^x\cos y$. At $P=(0,\pi/2)$: $g_x=1\cdot1=1$, $g_y=1\cdot0=0$, so $\nabla g=(1,0)$ (2). Unit vector of $(3,4)$: magnitude $5$, $\hat v=(3/5,4/5)$ (1.5). $D_{\hat v}g = \nabla g\cdot\hat v = 1\cdot\tfrac35 + 0 = \tfrac35 = 0.6$ (1.5).

(b) (5) Steepest ascent is along $\nabla g/\|\nabla g\| = (1,0)$ (unit vector) (2.5). Maximum rate $=\|\nabla g\| = \sqrt{1^2+0^2}=1$ (2.5).

Question 3 (14 marks)

(a) (4) Lagrangian: $\mathcal L = xy - \lambda(x^2+2y^2-6).$ Stationarity:

• $\partial_x: y - 2\lambda x = 0$ (1.5)
• $\partial_y: x - 4\lambda y = 0$ (1.5)
• constraint: $x^2+2y^2=6$ (1)

(b) (6) From eqs: $y=2\lambda x$, $x=4\lambda y = 8\lambda^2 x$. If $x\neq0$: $1=8\lambda^2 \Rightarrow \lambda=\pm\tfrac{1}{2\sqrt2}$ (2). Then $y=2\lambda x=\pm\tfrac{x}{\sqrt2}$, i.e. $y^2 = x^2/2$ (1). Constraint: $x^2+2(x^2/2)=6 \Rightarrow 2x^2=6\Rightarrow x^2=3\Rightarrow x=\pm\sqrt3$ (1.5). Then $y=\pm\sqrt{3/2}=\pm\sqrt{1.5}$. Four candidate points: $(\sqrt3,\sqrt{1.5}),\ (\sqrt3,-\sqrt{1.5}),\ (-\sqrt3,\sqrt{1.5}),\ (-\sqrt3,-\sqrt{1.5}).$ (1.5) (If $x=0$ then $y=0$, which violates constraint — rejected.)

(c) (4) $f=xy=\pm\sqrt3\cdot\sqrt{1.5}=\pm\sqrt{4.5}=\pm\tfrac{3}{\sqrt2}\approx\pm2.121$.

• Maximum $f=\tfrac{3}{\sqrt2}$ at $(\sqrt3,\sqrt{1.5})$ and $(-\sqrt3,-\sqrt{1.5})$ (same-sign product). (2)
• Minimum $f=-\tfrac{3}{\sqrt2}$ at the opposite-sign pairs. (2)

Question 4 (12 marks)

(a) (4) $h(x)=\ln(1+e^x)$. $h'(x)=\dfrac{e^x}{1+e^x}=\sigma(x)$ (sigmoid) (2). $h''(x)=\dfrac{e^x(1+e^x)-e^x\cdot e^x}{(1+e^x)^2}=\dfrac{e^x}{(1+e^x)^2}=\sigma(x)(1-\sigma(x))$ (2).

(b) (4) At $x=0$: $h(0)=\ln 2$; $h'(0)=\tfrac12$; $h''(0)=\tfrac12\cdot\tfrac12=\tfrac14$. $\tilde h(x)=\ln2 + \tfrac12 x + \tfrac12\cdot\tfrac14 x^2 = \ln 2 + \tfrac{x}{2} + \tfrac{x^2}{8}.$ (1 each for the three coefficients; 1 for correct $\frac12$ factor on quadratic term.)

(c) (4) $h''(x)=\dfrac{e^x}{(1+e^x)^2}$. For all real $x$, $e^x>0$ and $(1+e^x)^2>0$, so $h''(x)>0$ everywhere (3). A twice-differentiable function with strictly positive second derivative is (strictly) convex on $\mathbb{R}$ (1).

Question 5 (10 marks)

(a) (6) Chain rule. $\dfrac{\partial L}{\partial a}=(a-t)$; $\dfrac{da}{du}=1-\tanh^2(u)=1-a^2$; $\dfrac{\partial u}{\partial w}=x$, $\dfrac{\partial u}{\partial b}=1$.

$\frac{\partial L}{\partial w}=(a-t)(1-a^2)\,x,\qquad \frac{\partial L}{\partial b}=(a-t)(1-a^2).$

• $\partial L/\partial a$ (1), $da/du$ (1.5), assembling $\partial L/\partial w$ with the $x$ factor (2), $\partial L/\partial b$ (1.5).

(b) (4) (i) Too large: updates overshoot the minimum, loss oscillates or diverges (weights blow up) — no convergence (2). (ii) Too small: steps are tiny, convergence is very slow and may stall / need many iterations; can get stuck near flat regions (2).

[
  {"claim":"Gradient of f at (1,0) is (3,0)","code":"x,y=symbols('x y'); f=x**3-3*x*y**2+y**4; g=[diff(f,x),diff(f,y)]; result = [gg.subs({x:1,y:0}) for gg in g]==[3,0]"},
  {"claim":"(3/2,3/2) is a critical point of f","code":"x,y=symbols('x y'); f=x**3-3*x*y**2+y**4; result = (diff(f,x).subs({x:Rational(3,2),y:Rational(3,2)})==0) and (diff(f,y).subs({x:Rational(3,2),y:Rational(3,2)})==0)"},
  {"claim":"Directional derivative of e^x sin y at (0,pi/2) along (3,4) equals 3/5","code":"x,y=symbols('x y'); g=exp(x)*sin(y); gx=diff(g,x).subs({x:0,y:pi/2}); gy=diff(g,y).subs({x:0,y:pi/2}); dv=gx*Rational(3,5)+gy*Rational(4,5); result = simplify(dv-Rational(3,5))==0"},
  {"claim":"Max of xy on x^2+2y^2=6 is 3/sqrt(2)","code":"val = sqrt(3)*sqrt(Rational(3,2)); result = simplify(val - 3/sqrt(2))==0"},
  {"claim":"softplus second-order Taylor coeffs are ln2, 1/2, 1/8","code":"x=symbols('x'); h=log(1+exp(x)); s=series(h,x,0,3).removeO(); result = (s.coeff(x,0)==log(2)) and (s.coeff(x,1)==Rational(1,2)) and (s.coeff(x,2)==Rational(1,8))"},
  {"claim":"softplus second derivative positive at x=0","code":"x=symbols('x'); h=log(1+exp(x)); result = diff(h,x,2).subs(x,0) > 0"}
]