Noesis
Level 2 — RecallModel Evaluation & Selection

Model Evaluation & Selection

⏱ 30 minutes40 marksprintable — key stays hidden on paper

Subject: AI-ML
Level: 2 (Recall / Standard textbook problems)
Time limit: 30 minutes
Total marks: 40

Q1. Define the bias-variance tradeoff. State how bias and variance each change as model complexity increases. (4 marks)

Q2. A model achieves 98% training accuracy but only 72% validation accuracy. Diagnose whether this is underfitting or overfitting, and give two remedies. (4 marks)

Q3. For a binary classifier, the confusion matrix on a test set is:

Predicted Positive Predicted Negative
Actual Positive 40 (TP) 10 (FN)
Actual Negative 20 (FP) 30 (TN)

Compute accuracy, precision, recall, and F1-score. (6 marks)

Q4. Explain k-fold cross-validation in 3–4 sentences. State one advantage over a single train/validation split. (4 marks)

Q5. A regression model produces predictions for 4 samples:

True yy 3 5 2 8
Pred y^\hat{y} 2 5 4 6

Compute MAE, MSE, and RMSE. (6 marks)

Q6. Define stratified k-fold cross-validation and state one situation where it is preferred over ordinary k-fold. (3 marks)

Q7. Briefly describe the ROC curve and what AUC represents. What AUC value corresponds to a random classifier? (4 marks)

Q8. Compare grid search and random search for hyperparameter tuning. Give one advantage of each. (4 marks)

Q9. Write the formula for binary log-loss (cross-entropy) for a single sample with true label y∈{0,1}y \in \{0,1\} and predicted probability pp. Compute the log-loss when y=1y=1, p=0.8p=0.8. (5 marks)

Answer keyMark scheme & solutions

Q1. (4 marks)

  • Definition (2): The bias-variance tradeoff describes the tension between two sources of error: bias (error from overly simplistic assumptions, causing systematic underfitting) and variance (error from sensitivity to training-data fluctuations, causing overfitting). Total expected error = bias² + variance + irreducible noise. Why: minimizing one tends to increase the other, so we seek a balance.
  • As complexity increases (2): Bias decreases (1), variance increases (1).

Q2. (4 marks)

  • Diagnosis (2): Overfitting — large gap between high training accuracy and lower validation accuracy indicates the model memorizes training data but generalizes poorly.
  • Remedies (2, any two × 1): regularization (L1/L2), more training data, reduce model complexity, dropout, early stopping, data augmentation.

Q3. (6 marks)
TP=40, FN=10, FP=20, TN=30; total=100.

  • Accuracy = (TP+TN)/total = (40+30)/100 = 0.70 (1.5)
  • Precision = TP/(TP+FP) = 40/60 = 0.6667 (1.5)
  • Recall = TP/(TP+FN) = 40/50 = 0.80 (1.5)
  • F1 = 2·P·R/(P+R) = 2(0.6667·0.8)/(0.6667+0.8) = 1.0667/1.4667 = 0.7273 (1.5)

Why: precision = correctness of positive predictions; recall = coverage of actual positives; F1 = harmonic mean balancing both.

Q4. (4 marks)

  • Explanation (3): Data is split into k equal folds. The model is trained on k−1 folds and validated on the held-out fold, repeated k times so each fold serves once as validation. The performance is averaged over all k runs.
  • Advantage (1): Uses all data for both training and validation → lower-variance, more reliable performance estimate than a single split.

Q5. (6 marks)
Errors ∣y−y^∣|y-\hat y|: |3−2|=1, |5−5|=0, |2−4|=2, |8−6|=2.

  • MAE = (1+0+2+2)/4 = 5/4 = 1.25 (2)
  • MSE = (1²+0²+2²+2²)/4 = (1+0+4+4)/4 = 9/4 = 2.25 (2)
  • RMSE = √2.25 = 1.5 (2)

Why: MAE averages absolute errors (robust); MSE penalizes large errors more; RMSE restores original units.

Q6. (3 marks)

  • Definition (2): Stratified k-fold preserves the class proportion (label distribution) in each fold to match the overall dataset.
  • When preferred (1): With imbalanced classes (or small datasets), ensuring each fold has representative class ratios.

Q7. (4 marks)

  • ROC curve (2): Plots True Positive Rate (recall) vs False Positive Rate as the classification threshold varies.
  • AUC (1): Area Under the ROC Curve; probability that a random positive is ranked above a random negative — measures ranking/discrimination ability.
  • Random classifier (1): AUC = 0.5.

Q8. (4 marks)

  • Grid search (1+1): Exhaustively tries all combinations of a predefined grid. Advantage: systematic, guaranteed to test every specified point.
  • Random search (1+1): Samples random combinations from the search space. Advantage: more efficient in high dimensions / when few hyperparameters matter, better coverage per budget.

Q9. (5 marks)

  • Formula (3):   L=−[ylog⁡p+(1−y)log⁡(1−p)]\;L = -\big[y\log p + (1-y)\log(1-p)\big]
  • Computation (2): y=1,p=0.8⇒L=−log⁡(0.8)=0.2231y=1,p=0.8 \Rightarrow L = -\log(0.8) = 0.2231 (natural log). (Accept 0.3219 if log base 2.)

Why: log-loss heavily penalizes confident wrong predictions and rewards well-calibrated probabilities.

[
  {"claim":"Q3 metrics accuracy/precision/recall/F1 correct","code":"TP,FN,FP,TN=40,10,20,30; acc=(TP+TN)/100; prec=TP/(TP+FP); rec=TP/(TP+FN); f1=2*prec*rec/(prec+rec); result = abs(acc-0.70)<1e-9 and abs(prec-2/3)<1e-9 and abs(rec-0.80)<1e-9 and abs(f1-0.72727272727)<1e-6"},
  {"claim":"Q5 MAE=1.25, MSE=2.25, RMSE=1.5","code":"import sympy as sp; y=[3,5,2,8]; yh=[2,5,4,6]; e=[abs(a-b) for a,b in zip(y,yh)]; mae=sum(e)/4; mse=sum(t**2 for t in e)/4; rmse=sp.sqrt(mse); result = mae==sp.Rational(5,4) and mse==sp.Rational(9,4) and rmse==sp.Rational(3,2)"},
  {"claim":"Q9 log-loss for y=1,p=0.8 equals -ln(0.8)=0.2231","code":"import sympy as sp; L=-sp.log(0.8); result = abs(float(L)-0.22314355)<1e-6"}
]