Model Evaluation & Selection

Question 1 (12 marks)

(a) TP=45, FN=15, FP=20, TN=120. Total=200. (1 for reading values)

• Accuracy $= (TP+TN)/200 = 165/200 = 0.825$ (1)
• Precision $= TP/(TP+FP) = 45/65 = 0.6923$ (2)
• Recall $= TP/(TP+FN) = 45/60 = 0.75$ (1)
• $F_1 = 2PR/(P+R) = 2(0.6923)(0.75)/(0.6923+0.75) = 1.0385/1.4423 = 0.7200$ (1)

(b) $F_1$ is the harmonic mean of $P$ and $R$: (1) $F_1 = \frac{2}{\frac{1}{P}+\frac{1}{R}} = \frac{2}{\frac{P+R}{PR}} = \frac{2PR}{P+R}$ (2)

(c) A trivial "all negative" classifier scores $140/200 = 70\%$ accuracy while catching zero positives. (1) With imbalance, accuracy is dominated by the majority class and hides poor positive detection. (1) Report $F_1$ (or recall if false negatives are costly) since it balances precision and recall on the minority class. (1)

Question 2 (11 marks)

(a) Assume $y = f(x) + \varepsilon$ with $\mathbb{E}[\varepsilon]=0$, $\text{Var}(\varepsilon)=\sigma^2$, noise independent of model. (1)

Let $\bar{f} = \mathbb{E}[\hat{f}(x)]$. Then: $\mathbb{E}[(y-\hat f)^2] = \mathbb{E}[(f + \varepsilon - \hat f)^2]$ (1)

Expand around $\bar f$: add and subtract $\bar f$: (1) $= \mathbb{E}[(f - \bar f)^2] + \mathbb{E}[(\bar f - \hat f)^2] + \mathbb{E}[\varepsilon^2] + \text{cross terms}$

Cross terms vanish: $\mathbb{E}[\varepsilon]=0$ and $\mathbb{E}[\bar f - \hat f]=0$, noise independent. (2) $= \underbrace{(f-\bar f)^2}_{\text{Bias}^2} + \underbrace{\mathbb{E}[(\hat f - \bar f)^2]}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{irreducible}}$ (2)

(b) Large gap (4% vs 22%) ⇒ high variance / overfitting. (2) Remedies (any two): more training data, stronger regularisation, reduce model complexity, feature selection, early stopping. (2)

Question 3 (12 marks)

(a) (6: shuffle 1, split logic 3, correct tuples 2)

import numpy as np
def kfold_indices(n, k, seed=0):
    rng = np.random.default_rng(seed)
    idx = rng.permutation(n)
    folds = np.array_split(idx, k)
    result = []
    for i in range(k):
        val = folds[i]
        train = np.concatenate([folds[j] for j in range(k) if j != i])
        result.append((train, val))
    return result

(b) Stratified k-fold preserves the class proportions in every fold. (2) Matters for imbalanced data because random splitting can produce folds with few/no minority samples, giving high-variance or invalid metric estimates. (1)

(c) Nested CV = outer loop estimates generalisation error; inner loop does hyperparameter selection on each outer training partition. (2) Avoids the pitfall of optimistic bias / data leakage from tuning hyperparameters on the same data used to report performance. (1)

Question 4 (10 marks)

(a) AUC = fraction of positive–negative pairs where positive scores higher. 4×4 = 16 pairs. (1)

Count discordant/tied pairs. Negatives: {0.7,0.5,0.4,0.2}.

• Pos 0.9 > all 4 → 4 (concordant)
• Pos 0.8 > all 4 → 4
• Pos 0.6 > {0.5,0.4,0.2} = 3; loses to 0.7 → 3
• Pos 0.35 > {0.2} = 1; loses to 0.7,0.5,0.4 → 1

Concordant = 4+4+3+1 = 12. (4) $\text{AUC} = 12/16 = 0.75$ (1)

(b) AUC = 0.5 means the model ranks no better than random. (1) AUC preferable to accuracy when classes are imbalanced or a decision threshold isn't fixed (threshold-independent ranking quality). (1)

(c) ROC plots TPR vs FPR; PR plots precision vs recall. (1) Prefer PR when the positive class is rare, because ROC can look optimistic under heavy imbalance while PR focuses on positive-class performance. (1)

Question 5 (15 marks)

(a) Errors $\hat y - y$: $[0.5, 0, -1.5, -1]$. (1)

• MAE $= (0.5+0+1.5+1)/4 = 3/4 = 0.75$ (2)
• MSE $= (0.25+0+2.25+1)/4 = 3.5/4 = 0.875$ (2)
• RMSE $= \sqrt{0.875} = 0.9354$ (1)

(b) RMSE squares errors before averaging, so a large error contributes quadratically; a single big miss inflates RMSE far more than MAE (which is linear in error). (3)

(c) $\text{LogLoss} = -\frac{1}{N}\sum [y\ln\hat p + (1-y)\ln(1-\hat p)]$. (2)

Terms: $-\ln 0.9 = 0.10536$; $-\ln(1-0.2)=-\ln 0.8 = 0.22314$; $-\ln 0.7 = 0.35667$. (1) $\text{LogLoss} = (0.10536+0.22314+0.35667)/3 = 0.68517/3 = 0.2284$ (1)

(d) Calibration: predicted probabilities match observed frequencies (of predicted-0.7 events, ~70% are positive). (1) Reliability diagram: bin predictions by probability, plot mean predicted prob (x) vs empirical positive fraction (y); diagonal = perfectly calibrated. (1)

[
{"claim":"F1 = 0.72 from P=45/65, R=45/60","code":"P=Rational(45,65); R=Rational(45,60); F1=2*P*R/(P+R); result=(abs(float(F1)-0.72)<1e-9)"},
{"claim":"Accuracy = 0.825","code":"result=(Rational(165,200)==Rational(33,40))"},
{"claim":"AUC = 0.75 from 12 concordant of 16","code":"result=(Rational(12,16)==Rational(3,4))"},
{"claim":"MAE=0.75, MSE=0.875, RMSE=sqrt(0.875)","code":"errs=[0.5,0,-1.5,-1]; mae=sum(abs(e) for e in errs)/4; mse=sum(e**2 for e in errs)/4; import math; rmse=math.sqrt(mse); result=(abs(mae-0.75)<1e-9 and abs(mse-0.875)<1e-9 and abs(rmse-0.9354143)<1e-6)"},
{"claim":"LogLoss approx 0.2284","code":"import math; ll=(-math.log(0.9)-math.log(0.8)-math.log(0.7))/3; result=(abs(ll-0.22839)<1e-4)"}
]