Noesis
Level 2 — RecallMachine Learning (Aerospace Applications)

Machine Learning (Aerospace Applications)

⏱ 30 minutes40 marksprintable — key stays hidden on paper

Difficulty: Level 2 — Recall (definitions, standard problems, short derivations) Time limit: 30 minutes Total marks: 40

Answer all questions. Show working where derivations are asked. Use ...... for math.

Q1. (4 marks) Write the normal equation solution for linear regression with design matrix XX and target vector yy. Given X=[111213],y=[122],X = \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}, \quad y = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix}, compute the fitted parameter vector θ=(XTX)−1XTy\theta = (X^TX)^{-1}X^Ty.

Q2. (4 marks) Define the sigmoid function σ(z)\sigma(z) and show that its derivative satisfies σ′(z)=σ(z)(1−σ(z))\sigma'(z) = \sigma(z)(1-\sigma(z)).

Q3. (4 marks) State the binary cross-entropy loss for a single example with true label y∈{0,1}y\in\{0,1\} and predicted probability p^\hat{p}. Evaluate it for y=1y=1, p^=0.8\hat{p}=0.8 (use natural log).

Q4. (4 marks) Explain the difference between L1 (lasso) and L2 (ridge) regularization. State one practical consequence of each on the learned weights.

Q5. (4 marks) Define the bias–variance trade-off. State qualitatively how increasing model complexity affects bias and variance.

Q6. (5 marks) Perform one forward pass through a single neuron with inputs x=[2,−1]x=[2,-1], weights w=[0.5,1.0]w=[0.5, 1.0], bias b=0.5b=0.5, and ReLU activation. Then repeat with a sigmoid activation (leave the sigmoid value as σ(⋅)\sigma(\cdot) and give a decimal).

Q7. (4 marks) Briefly explain k-fold cross-validation. For a dataset of 100 samples with k=5k=5, how many samples are used for training and validation in each fold?

Q8. (4 marks) Write the SGD parameter update rule with learning rate η\eta and gradient gtg_t. Then write the momentum update (velocity form) with momentum coefficient β\beta.

Q9. (4 marks) Given query qq, keys KK, and values VV, write the scaled dot-product attention formula and explain why the scaling factor dk\sqrt{d_k} is used.

Q10. (3 marks) In a Markov Decision Process, state the Bellman optimality equation for the state-value function V∗(s)V^*(s).

Answer keyMark scheme & solutions

Q1. (4 marks) Normal equation: θ=(XTX)−1XTy\theta = (X^TX)^{-1}X^Ty. (1)

Compute: XTX=[36614],XTy=[511].X^TX = \begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix}, \quad X^Ty = \begin{bmatrix} 5 \\ 11 \end{bmatrix}. (1)

det⁡=3⋅14−36=6\det = 3\cdot14 - 36 = 6, so (XTX)−1=16[14−6−63](X^TX)^{-1} = \frac{1}{6}\begin{bmatrix} 14 & -6 \\ -6 & 3 \end{bmatrix}. (1)

θ=16[14−6−63][511]=16[70−66−30+33]=[2/31/2].\theta = \frac{1}{6}\begin{bmatrix} 14 & -6 \\ -6 & 3 \end{bmatrix}\begin{bmatrix} 5 \\ 11 \end{bmatrix} = \frac{1}{6}\begin{bmatrix} 70-66 \\ -30+33 \end{bmatrix} = \begin{bmatrix} 2/3 \\ 1/2 \end{bmatrix}. (1)

So θ0=2/3≈0.667\theta_0 = 2/3 \approx 0.667, θ1=1/2=0.5\theta_1 = 1/2 = 0.5.

Q2. (4 marks) σ(z)=11+e−z\sigma(z) = \dfrac{1}{1+e^{-z}}. (1) σ′(z)=ddz(1+e−z)−1=−(1+e−z)−2⋅(−e−z)=e−z(1+e−z)2\sigma'(z) = \dfrac{d}{dz}(1+e^{-z})^{-1} = -(1+e^{-z})^{-2}\cdot(-e^{-z}) = \dfrac{e^{-z}}{(1+e^{-z})^2}. (1) Write e−z(1+e−z)2=11+e−z⋅e−z1+e−z=σ(z)⋅(1−σ(z))\dfrac{e^{-z}}{(1+e^{-z})^2} = \dfrac{1}{1+e^{-z}}\cdot\dfrac{e^{-z}}{1+e^{-z}} = \sigma(z)\cdot(1-\sigma(z)) since e−z1+e−z=1−σ(z)\dfrac{e^{-z}}{1+e^{-z}} = 1-\sigma(z). (2)

Q3. (4 marks) L=−[yln⁡p^+(1−y)ln⁡(1−p^)]L = -[y\ln\hat{p} + (1-y)\ln(1-\hat{p})]. (2) For y=1y=1, p^=0.8\hat{p}=0.8: L=−ln⁡(0.8)≈0.2231L = -\ln(0.8) \approx 0.2231. (2)

Q4. (4 marks)

  • L1 (lasso): penalty λ∑∣wi∣\lambda\sum|w_i|; produces sparse weights (drives some to exactly zero → feature selection). (2)
  • L2 (ridge): penalty λ∑wi2\lambda\sum w_i^2; shrinks weights smoothly toward zero but rarely exactly zero; distributes weight among correlated features. (2)

Q5. (4 marks) Total expected error decomposes into (bias)2^2 + variance + irreducible noise. (1) Bias = error from wrong model assumptions (underfitting); variance = sensitivity to training data (overfitting). (1) Increasing complexity decreases bias (1) and increases variance (1); optimal complexity balances the two.

Q6. (5 marks) Pre-activation z=w⋅x+b=0.5(2)+1.0(−1)+0.5=1−1+0.5=0.5z = w\cdot x + b = 0.5(2) + 1.0(-1) + 0.5 = 1 - 1 + 0.5 = 0.5. (2) ReLU: max⁡(0,0.5)=0.5\max(0, 0.5) = 0.5. (1) Sigmoid: σ(0.5)=11+e−0.5≈0.6225\sigma(0.5) = \dfrac{1}{1+e^{-0.5}} \approx 0.6225. (2)

Q7. (4 marks) k-fold CV: split data into kk equal folds; train on k−1k-1 folds, validate on the held-out fold, rotate so each fold is validation once; average the kk scores. (2) For 100 samples, k=5k=5: each fold has 20 samples → 80 training, 20 validation per fold. (2)

Q8. (4 marks) SGD: θt+1=θt−ηgt\theta_{t+1} = \theta_t - \eta g_t. (2) Momentum: vt=βvt−1+gtv_t = \beta v_{t-1} + g_t (or (1−β)gt(1-\beta)g_t); θt+1=θt−ηvt\theta_{t+1} = \theta_t - \eta v_t. (2)

Q9. (4 marks) Attention(Q,K,V)=softmax ⁣(QKTdk)V\text{Attention}(Q,K,V) = \text{softmax}\!\left(\dfrac{QK^T}{\sqrt{d_k}}\right)V. (2) Scaling by dk\sqrt{d_k} prevents dot products from growing large in magnitude with dimension dkd_k, which would push softmax into regions of very small gradients (saturation); it keeps variance controlled. (2)

Q10. (3 marks) V∗(s)=max⁡a∑s′P(s′∣s,a)[R(s,a,s′)+γV∗(s′)].V^*(s) = \max_a \sum_{s'} P(s'|s,a)\big[R(s,a,s') + \gamma V^*(s')\big]. (3) (Equivalent forms with expectation notation accepted.)

[
  {"claim":"Normal equation gives theta = [2/3, 1/2]","code":"X=Matrix([[1,1],[1,2],[1,3]]); y=Matrix([1,2,2]); th=(X.T*X).inv()*X.T*y; result = (th == Matrix([Rational(2,3), Rational(1,2)]))"},
  {"claim":"Cross-entropy for y=1,p=0.8 is approx 0.2231","code":"L=-log(0.8); result = abs(float(L)-0.2231)<1e-3"},
  {"claim":"Neuron pre-activation z=0.5 and sigmoid approx 0.6225","code":"z=0.5*2+1.0*(-1)+0.5; s=1/(1+exp(-z)); result = (z==0.5) and abs(float(s)-0.6225)<1e-3"},
  {"claim":"Sigmoid derivative equals s*(1-s)","code":"z=symbols('z'); s=1/(1+exp(-z)); result = simplify(diff(s,z) - s*(1-s))==0"}
]