Machine Learning (Aerospace Applications)

Question 1

(a) [6] Gradient: $\nabla J = X^\top(X\theta - y) + \lambda\theta = 0$ (2). Rearranging: $(X^\top X + \lambda I)\theta = X^\top y$ (1), hence $\theta^\star = (X^\top X + \lambda I)^{-1}X^\top y.$ (1) Uniqueness: $X^\top X$ is symmetric PSD with eigenvalues $\mu_i\ge 0$. Adding $\lambda I$ shifts eigenvalues to $\mu_i+\lambda>0$ for $\lambda>0$, so $X^\top X+\lambda I$ is positive-definite, hence invertible; $J$ is strictly convex ⇒ unique minimiser (2).

(b) [4] $X^\top X = n$, $X^\top y = \sum y_i = n\bar y$ (2). Then $\theta^\star=\frac{n\bar y}{n+\lambda}.$ (1) As $\lambda\to0$, $\theta^\star\to\bar y$ (OLS mean); as $\lambda\to\infty$, $\theta^\star\to0$ — shrinkage toward zero, reducing variance at the cost of bias (1).

(c) [6] $\mathbb E[(y-\hat f)^2] = \sigma^2 + (\text{Bias}[\hat f])^2 + \text{Var}[\hat f]$ (3) where $\sigma^2$ irreducible noise, Bias $=\mathbb E[\hat f]-f$, Var $=\mathbb E[(\hat f-\mathbb E\hat f)^2]$. Increasing $\lambda$ constrains parameters → less sensitivity to noise in flight data → variance ↓ but model less flexible → bias ↑ (2). Sketch: bias$^2$ increasing curve, variance decreasing curve, total error U-shaped with minimum at optimal $\lambda$ (1).

(d) [4] 5-fold ⇒ 5 models trained per $\lambda$ value (1). Each fold: train on $80$ samples, validate on $20$ (2). L1 induces sparsity (drives irrelevant weights to exactly zero) performing feature selection when only a few sensors matter (1).

Question 2

(a) [5] $\sigma'(z)=\sigma(z)(1-\sigma(z))=\hat p(1-\hat p)$ (1). $\dfrac{\partial L}{\partial\hat p} = -\dfrac{y}{\hat p}+\dfrac{1-y}{1-\hat p} = \dfrac{\hat p - y}{\hat p(1-\hat p)}$ (2). Chain: $\dfrac{\partial L}{\partial z^{(2)}}=\dfrac{\partial L}{\partial\hat p}\cdot\hat p(1-\hat p)=\hat p - y$ (2).

(b) [7] Let $\delta_2=\hat p-y$ (scalar). $\dfrac{\partial L}{\partial W_2}=\delta_2\,h^\top$ (2). Backprop to hidden: $\delta_1 = (W_2^\top\delta_2)\odot(1-\tanh^2 z^{(1)})=(W_2^\top\delta_2)\odot(1-h^2)$ (3). $\dfrac{\partial L}{\partial W_1}=\delta_1 x^\top$ (1); $\dfrac{\partial L}{\partial b_1}=\delta_1$ (1).

(c) [6] (i) $z^{(1)}=0.5(1)+(-0.5)(2)=0.5-1.0=-0.5$ (1). $h=\tanh(-0.5)=-0.46212$ (1). $z^{(2)}=1.0\cdot(-0.46212)=-0.46212$; $\hat p=\sigma(-0.46212)=0.38648$ (2). (ii) $\partial L/\partial z^{(2)}=\hat p-y=0.38648-1=-0.61352$ (1). (iii) $\partial L/\partial W_2=\delta_2 h=(-0.61352)(-0.46212)=0.28352$ (1).

(d) [4] $\tanh'$ saturates to $0$ for large $|z|$ ⇒ vanishing gradients in deep stacks slowing learning (2). ReLU has gradient $1$ for $z>0$, avoiding saturation and giving sparse activations (1). Drawback: "dying ReLU" — units stuck at $0$ with zero gradient (or non-differentiable at 0) (1).

Question 3

(a) [6] $Q^\star(s,a)=\mathbb E\big[R + \gamma\max_{a'}Q^\star(s',a')\mid s,a\big]$ (2). Sample-based stochastic approximation gives update: $Q(s,a)\leftarrow Q(s,a)+\alpha\big[r+\gamma\max_{a'}Q(s',a')-Q(s,a)\big].$ (2) $\alpha$ weights new TD-target vs. current estimate (controls learning speed/stability) (1). Off-policy: the update uses $\max_{a'}$ (greedy target) regardless of the behaviour policy that generated the action (1).

(b) [7] $J(\theta)=\mathbb E_\tau[R(\tau)]=\sum_\tau P_\theta(\tau)R(\tau)$. $\nabla_\theta J=\sum_\tau \nabla_\theta P_\theta(\tau)R(\tau)=\sum_\tau P_\theta(\tau)\nabla_\theta\ln P_\theta(\tau)R(\tau)$ (log-trick) (3). $\ln P_\theta(\tau)=\sum_t\ln\pi_\theta(a_t|s_t)+\text{(dynamics terms independent of }\theta)$, so $\nabla_\theta\ln P_\theta=\sum_t\nabla_\theta\ln\pi_\theta(a_t|s_t)$ (2). Replacing $R(\tau)$ with return-to-go $G_t$ gives the stated form (1). Baseline unbiased: $\mathbb E[\nabla_\theta\ln\pi_\theta(a|s)\,b(s)]=b(s)\sum_a\nabla_\theta\pi_\theta(a|s)=b(s)\nabla_\theta 1=0$ (1).

(c) [5] Target $=r+\gamma\max_{a'}Q(\text{term},\cdot)=5+0.9(0)=5$ (1). Update: $Q\leftarrow 0+0.5(5-0)=2.5$ (2). Converged (deterministic, terminal next): fixed point $Q=r+\gamma\cdot0=5$ (2).

[
  {"claim":"Ridge scalar all-ones estimate n*ybar/(n+lambda)", "code":"n,lam=5,3; ys=[1,2,3,4,5]; ybar=sum(ys)/n; theta=n*ybar/(n+lam); result = abs(theta - (n*ybar/(n+lam)))<1e-12 and abs(theta-2.8125)<1e-9"},
  {"claim":"Q2c forward/backward numeric values", "code":"import sympy as sp; z1=sp.Rational(1,2)*1 + sp.Rational(-1,2)*2; h=sp.tanh(z1); z2=1*h; phat=1/(1+sp.exp(-z2)); d2=phat-1; gW2=d2*h; result = abs(float(z1)+0.5)<1e-9 and abs(float(h)+0.46211715)<1e-6 and abs(float(phat)-0.38648)<1e-4 and abs(float(gW2)-0.28352)<1e-4"},
  {"claim":"Q-learning single update yields 2.5 and converges to 5", "code":"alpha,gamma,r=0.5,0.9,5; Q=0; Q=Q+alpha*(r+gamma*0-Q); conv=r+gamma*0; result = abs(Q-2.5)<1e-12 and abs(conv-5)<1e-12"},
  {"claim":"Sigmoid derivative identity used in Q2a", "code":"import sympy as sp; z=sp.symbols('z'); s=1/(1+sp.exp(-z)); result = sp.simplify(sp.diff(s,z) - s*(1-s))==0"}
]