Calculus & Optimization Basics

Question 1 (12)

1. Gradient (2) $\nabla f = \left(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}\right) = (3x^2 - 3y,\; -3x + 3y^2)$ (1 mark each partial.)

2. Critical points (3) Set both to zero: $x^2 = y$ and $y^2 = x$. Substitute $y=x^2$ into $y^2=x$: $x^4 = x \Rightarrow x(x^3-1)=0 \Rightarrow x=0$ or $x=1$.

• $x=0 \Rightarrow y=0$: point $(0,0)$.
• $x=1 \Rightarrow y=1$: point $(1,1)$. (1 for equations, 1 each point.)

3. Hessian (2) $H = \begin{pmatrix} 6x & -3 \\ -3 & 6y \end{pmatrix}$

4. Classification (5) Det $= 36xy - 9$.

• At $(0,0)$: $\det = -9 < 0 \Rightarrow$ saddle point. (2)
• At $(1,1)$: $\det = 36-9 = 27 > 0$, and leading minor $f_{xx}=6>0 \Rightarrow$ local minimum. (2)

Explain out loud (1): The second-derivative test uses the Hessian as a local quadratic model. A negative determinant means the two curvature eigenvalues have opposite signs → surface curves up in one direction and down in another → saddle. A positive determinant with positive $f_{xx}$ means both eigenvalues positive → bowl → local min. The leading minor sign disambiguates min vs max when det>0.

Question 2 (10)

1. Taylor derivation (4) $g(0)=0$; $g'(x)=\frac{1}{1+x}$, $g'(0)=1$; $g''(x)=-\frac{1}{(1+x)^2}$, $g''(0)=-1$. $g(x) \approx 0 + 1\cdot x + \frac{-1}{2}x^2 = x - \frac{x^2}{2}$

2. Approximate $\ln 1.2$ (3) $x=0.2$: $0.2 - \frac{0.04}{2} = 0.2 - 0.02 = 0.18$. True $\ln 1.2 = 0.182321...$; absolute error $\approx 0.002321$. (2 for approx, 1 for error.)

3. Multivariate formula (3) $f(\mathbf{x}) \approx f(\mathbf{x}_0) + \nabla f(\mathbf{x}_0)^\top (\mathbf{x}-\mathbf{x}_0) + \tfrac{1}{2}(\mathbf{x}-\mathbf{x}_0)^\top H(\mathbf{x}_0)(\mathbf{x}-\mathbf{x}_0)$ Gradient term = linear (first-order slope); Hessian term = quadratic curvature correction.

Question 3 (12)

1. Update rule & recurrence (3) $f'(w) = (w-4)$. Update: $w_{t+1} = w_t - \eta f'(w_t) = w_t - \eta(w_t - 4)$. $w_{t+1} = (1-\eta)w_t + 4\eta$

2. Convergence condition (4) Error $e_t = w_t - 4$. Subtract 4: $e_{t+1} = (1-\eta)e_t$, so $e_t = (1-\eta)^t e_0$. Converges iff $|1-\eta| < 1 \Rightarrow 0 < \eta < 2$. (3) For $\eta > 2$: $|1-\eta|>1$ → error grows, diverges (oscillating with increasing amplitude). At $\eta=2$ it oscillates without decaying. (1)

3. Code (5)

import numpy as np
def gradient_descent(w0, eta, steps):
    w = w0
    for _ in range(steps):
        grad = w - 4          # f'(w)
        w = w - eta * grad
    return w

(2 loop, 1 correct gradient, 1 update, 1 return.)

Question 4 (10)

1. Lagrangian (4) $\mathcal{L} = x^2 + y^2 - \lambda(x + 2y - 5)$. Stationarity: $\partial_x: 2x - \lambda = 0,\quad \partial_y: 2y - 2\lambda = 0,\quad \partial_\lambda: x+2y-5=0$

2. Solve (4) From first two: $x = \lambda/2$, $y = \lambda$. Constraint: $\lambda/2 + 2\lambda = 5 \Rightarrow \tfrac{5\lambda}{2}=5 \Rightarrow \lambda = 2$. So $x=1,\ y=2$. Minimum value $f=1+4=5$.

3. Explain (2) $\lambda$ is the sensitivity of the optimal objective to relaxing the constraint (shadow price): $\frac{df^*}{dc}=\lambda$. At the optimum along the constraint, moving without violating $g$ cannot decrease $f$, which requires $\nabla f$ have no component tangent to the constraint curve — i.e. $\nabla f$ is parallel to $\nabla g$, giving $\nabla f=\lambda\nabla g$.

Question 5 (10)

1. Chain rule (6) Intermediates:

• $\frac{\partial L}{\partial a} = a - y$
• $\frac{\partial a}{\partial z} = \sigma(z)(1-\sigma(z)) = a(1-a)$
• $\frac{\partial z}{\partial w} = x$, $\frac{\partial z}{\partial b}=1$

$\frac{\partial L}{\partial w} = (a-y)\,a(1-a)\,x,\qquad \frac{\partial L}{\partial b} = (a-y)\,a(1-a)$ (2 per intermediate group + assembly.)

2. Numeric (4) $z = 0.5\cdot1 + 0 = 0.5$; $a = \sigma(0.5) = \frac{1}{1+e^{-0.5}} \approx 0.622459$. $(a-y) = 0.622459 - 1 = -0.377541$. $a(1-a) = 0.622459 \times 0.377541 \approx 0.234958$. $\frac{\partial L}{\partial w} = -0.377541 \times 0.234958 \times 1 \approx -0.088722$.

Question 6 (6)

1. Convexity (3) $f''(x)=2>0$ for all $x$ → convex. For a convex function the epigraph is a convex set and any local minimum satisfies $f(x)\ge f(x^*)$ everywhere, so the local minimum is global (no other lower valley can exist).

2. Directional derivative (3) $\nabla f = (2x, 6y) = (2,6)$ at $(1,1)$. Unit vector: $\|\mathbf{u}\|=5$, $\hat{\mathbf{u}}=(0.6,0.8)$. $D_{\hat u}f = \nabla f\cdot\hat u = 2(0.6)+6(0.8) = 1.2 + 4.8 = 6.0$.

[
  {"claim":"Q1 Hessian det at (1,1) is 27 -> local min", "code":"x,y=1,1; det=36*x*y-9; result=(det==27 and 6*x>0)"},
  {"claim":"Q2 Taylor approx of ln(1.2) = 0.18", "code":"x=Rational(2,10); approx=x-x**2/2; result=(approx==Rational(18,100))"},
  {"claim":"Q4 Lagrange solution x=1,y=2,lambda=2 satisfies constraint and stationarity", "code":"x,y,lam=1,2,2; result=(2*x-lam==0 and 2*y-2*lam==0 and x+2*y-5==0)"},
  {"claim":"Q5 dL/dw approx -0.08872", "code":"z=Rational(1,2); a=1/(1+exp(-z)); dLdw=(a-1)*a*(1-a)*1; result=(abs(float(dLdw)-(-0.088722))<1e-4)"},
  {"claim":"Q6 directional derivative equals 6.0", "code":"gx,gy=2,6; ux,uy=Rational(6,10),Rational(8,10); D=gx*ux+gy*uy; result=(D==6)"}
]