Level 4 — ApplicationModel Evaluation & Selection

Model Evaluation & Selection

⏱ 60 minutes60 marksprintable — key stays hidden on paper

Level 4 — Application (novel problems, no hints) Time limit: 60 minutes Total marks: 60

Q1. Confusion matrix & threshold economics (14 marks) A fraud-detection classifier is evaluated on 10,000 transactions. At the default threshold the confusion matrix is:

	Predicted Fraud	Predicted Legit
Actual Fraud	120	80
Actual Legit	300	9500

(a) Compute accuracy, precision, recall, and F1-score. (6) (b) The business estimates: a missed fraud (false negative) costs $500; a false alarm (false positive) costs $20 in review labour. Compute total expected cost at this threshold. (3) (c) Raising the threshold shifts 40 fraud cases from TP→FN and removes 200 false positives (FP→TN). Recompute total cost and state whether the new threshold is preferable, justifying with numbers. (5)

Q2. Cross-validation design under leakage (12 marks) You are given 900 medical records from 300 patients (3 records each). The target is disease severity. A colleague reports 96% accuracy using standard 10-fold CV with per-record shuffling, but the deployed model achieves only 71%.

(a) Explain the specific leakage mechanism causing the optimistic CV estimate. (4) (b) Describe a corrected CV scheme and state what the "fold unit" must be. (4) (c) The dataset is also class-imbalanced (10% severe). Explain what additional CV modification you would apply and why per-record stratification alone is insufficient here. (4)

Q3. Bias–variance and learning curves (12 marks) Two models are trained on the same regression task. Learning curves (error vs. training-set size) show:

Model A: training error plateaus at 0.30, validation error plateaus at 0.32; curves have converged.
Model B: training error 0.05, validation error 0.28; gap not closing as data grows.

(a) Diagnose each model as underfitting or overfitting, and locate each on the bias–variance spectrum. (4) (b) For each model, give two concrete, different remedies appropriate to its diagnosis. (4) (c) The irreducible (Bayes) error is estimated at 0.10. Decompose the expected test error conceptually into bias², variance, and noise, and state which term dominates for Model B. (4)

Q4. Regression metrics selection (10 marks) A demand-forecasting model produces the following (actual, predicted) pairs for 5 days: $(100, 90),\ (200, 260),\ (50, 55),\ (400, 380),\ (10, 40)$ .

(a) Compute MAE, RMSE, and MAPE. (6) (b) The client cares about percentage accuracy but one analyst warns MAPE is misleading here. Identify which data point drives the warning and explain why. (2) (c) State one situation where you would prefer RMSE over MAE and why. (2)

Q5. Probabilistic evaluation & ensembling (12 marks) Two binary classifiers output probabilities for the positive class on 4 samples with true labels $y=(1,0,1,0)$ :

Model P: $(0.9,\ 0.2,\ 0.6,\ 0.4)$
Model Q: $(0.7,\ 0.3,\ 0.8,\ 0.1)$

(a) Compute the average binary log-loss for Model Q. (Use natural log.) (4) (b) Build a soft-voting ensemble (average the two models' probabilities) and compute its log-loss. (4) (c) A stacking meta-learner is proposed instead of simple averaging. Explain how stacking differs from soft voting, and one pitfall (with its fix) when generating meta-features. (4)

Answer keyMark scheme & solutions

Q1 (14)

(a) TP=120, FN=80, FP=300, TN=9500.

Accuracy $=(120+9500)/10000 = 9620/10000 = 0.962$ (1)
Precision $=120/(120+300)=120/420=0.2857$ (2)
Recall $=120/(120+80)=120/200=0.60$ (2)
F1 $=2\cdot\frac{0.2857\cdot0.60}{0.2857+0.60}=2\cdot\frac{0.1714}{0.8857}=0.3871$ (1)

(b) Cost $=500\cdot FN + 20\cdot FP = 500\cdot80 + 20\cdot300 = 40000 + 6000 = \$ 46{,}000$. (3)

(c) New: FN $=80+40=120$ , FP $=300-200=100$ . Cost $=500\cdot120 + 20\cdot100 = 60000+2000 = \$ 62{,}000 $. (3)$ 62000 > 46000 $, so the new threshold is **worse** — the saved review cost (\$ 4,000) is dwarfed by the $20,000 extra from missed frauds. Keep the lower threshold. (2)

Q2 (12)

(a) Records from the same patient are correlated; per-record shuffling puts some of a patient's records in train and others in test. The model memorises patient-specific patterns, inflating CV accuracy — this is group/identity leakage. (4) (b) Use group-based (patient-grouped) K-fold: all 3 records of a patient stay in the same fold. The fold unit must be the patient, not the record. (4) (c) Apply stratified group K-fold so class proportions (10% severe) are preserved per fold; grouping alone can produce folds with almost no severe cases, giving high-variance/biased estimates. Must stratify at patient level while keeping groups intact. (4)

Q3 (12)

(a) Model A: converged with high train & val error → underfitting / high bias. Model B: low train, high val, persistent gap → overfitting / high variance. (4) (b) Model A (bias): increase model complexity/capacity; add features / reduce regularisation. Model B (variance): add training data / stronger regularisation / reduce complexity / feature selection (any two distinct each). (4) (c) Test error ≈ bias² + variance + noise; noise = irreducible ≈ 0.10. For Model B the large train–val gap (0.05→0.28) reflects a dominant variance term. (4)

Q4 (10)

Errors $(pred-act)$ : $-10, +60, +5, -20, +30$ ; abs: $10,60,5,20,30$ . MAE $=(10+60+5+20+30)/5 = 125/5 = 25$ . (2) RMSE: squares $100,3600,25,400,900$ ; sum $=5025$ ; mean $=1005$ ; $\sqrt{1005}=31.70$ . (2) MAPE: $|e|/act$ : $0.10,0.30,0.10,0.05,3.00$ ; sum $=3.55$ ; mean $=0.71 \Rightarrow 71\%$ . (2) (b) The point $(10,40)$ : error 30 on a true value of 10 = 300% — a small-magnitude actual inflates MAPE hugely; MAPE is unstable/biased for near-zero actuals. (2) (c) Prefer RMSE when large errors are disproportionately costly (it penalises them quadratically), e.g. safety-critical or outlier-sensitive forecasting. (2)

Q5 (12)

Log-loss per sample $=-[y\ln p + (1-y)\ln(1-p)]$ . (a) Model Q $(0.7,0.3,0.8,0.1)$ , $y=(1,0,1,0)$ :

$-\ln0.7=0.3567$
$-\ln0.7=0.3567$ (for $y=0,p=0.3$ : $-\ln0.7$ )
$-\ln0.8=0.2231$
$-\ln0.9=0.1054$ Sum $=1.0419$ ; mean $=0.2605$ . (4) (b) Ensemble probs = average of P,Q: $(0.8,0.25,0.7,0.25)$ .
$-\ln0.8=0.2231$
$-\ln0.75=0.2877$
$-\ln0.7=0.3567$
$-\ln0.75=0.2877$ Sum $=1.1552$ ; mean $=0.2888$ . (4) (c) Soft voting fixes equal (or preset) weights and averages probabilities; stacking trains a meta-learner on base-model outputs to learn optimal combination weights/nonlinearities. Pitfall: generating meta-features on the same data used to fit base models leaks information → over-optimistic meta-learner; fix with out-of-fold predictions (cross-validated stacking). (4)

[
{"claim":"Q1a precision recall F1","code":"TP,FN,FP,TN=120,80,300,9500\nprec=TP/(TP+FP); rec=TP/(TP+FN); f1=2*prec*rec/(prec+rec)\nresult = abs(prec-0.285714)<1e-4 and abs(rec-0.6)<1e-9 and abs(f1-0.387097)<1e-4"},
{"claim":"Q1 cost comparison","code":"c1=500*80+20*300; c2=500*120+20*100\nresult = c1==46000 and c2==62000 and c2>c1"},
{"claim":"Q4 MAE RMSE MAPE","code":"a=[100,200,50,400,10]; p=[90,260,55,380,40]\ne=[pi-ai for ai,pi in zip(a,p)]\nmae=sum(abs(x) for x in e)/5\nrmse=(sum(x*x for x in e)/5)**0.5\nmape=sum(abs(x)/ai for x,ai in zip(e,a))/5\nresult = abs(mae-25)<1e-9 and abs(rmse-31.7017)<1e-3 and abs(mape-0.71)<1e-9"},
{"claim":"Q5 ensemble log-loss","code":"import math\ny=[1,0,1,0]; P=[0.9,0.2,0.6,0.4]; Q=[0.7,0.3,0.8,0.1]\nens=[(pi+qi)/2 for pi,qi in zip(P,Q)]\ndef ll(y,pr): return -sum(yi*math.log(pi)+(1-yi)*math.log(1-pi) for yi,pi in zip(y,pr))/len(y)\nresult = abs(ll(y,Q)-0.26047)<1e-3 and abs(ll(y,ens)-0.28881)<1e-3"}
]