Noesis
Level 1 — RecognitionModel Evaluation & Selection

Model Evaluation & Selection

⏱ 20 minutes40 marksprintable — key stays hidden on paper

Time Limit: 20 minutes Total Marks: 40

Section A — Multiple Choice (1 mark each)

Choose the single best answer.

Q1. A model with high bias and low variance most likely suffers from:

  • A) Overfitting
  • B) Underfitting
  • C) Data leakage
  • D) Perfect fit

Q2. In k-fold cross-validation with k=5k=5 on 1000 samples, each fold used for validation contains approximately how many samples?

  • A) 100
  • B) 200
  • C) 500
  • D) 1000

Q3. Precision is defined as:

  • A) TPTP+FN\frac{TP}{TP+FN}
  • B) TPTP+FP\frac{TP}{TP+FP}
  • C) TP+TNTotal\frac{TP+TN}{\text{Total}}
  • D) TNTN+FP\frac{TN}{TN+FP}

Q4. The ROC curve plots:

  • A) Precision vs Recall
  • B) True Positive Rate vs False Positive Rate
  • C) Accuracy vs Threshold
  • D) TP vs TN

Q5. Leave-one-out cross-validation (LOOCV) on nn samples is equivalent to k-fold CV with:

  • A) k=1k=1
  • B) k=2k=2
  • C) k=nk=n
  • D) k=n/2k=n/2

Q6. Which metric is most appropriate for penalizing large errors more heavily in regression?

  • A) MAE
  • B) MSE
  • C) MAPE
  • D) Median error

Q7. An AUC of 0.5 indicates a classifier that:

  • A) Is perfect
  • B) Performs no better than random
  • C) Is inverted (worse than random)
  • D) Has zero false positives

Q8. Nested cross-validation is primarily used to:

  • A) Speed up training
  • B) Obtain an unbiased estimate of generalization when tuning hyperparameters
  • C) Reduce the dataset size
  • D) Eliminate the need for a test set

Q9. Stratified k-fold cross-validation ensures that:

  • A) Folds are the same size only
  • B) Class proportions are preserved in each fold
  • C) Samples are ordered by target value
  • D) Only the majority class is used

Q10. Log-loss heavily penalizes a prediction that is:

  • A) Correct and confident
  • B) Confident but wrong
  • C) Uncertain (probability near 0.5)
  • D) Missing entirely

Section B — Matching (2 marks each)

Q11. Match each metric to its formula.

Metric Formula
1. Recall A. TP+TNTP+TN+FP+FN\frac{TP+TN}{TP+TN+FP+FN}
2. Accuracy B. TPTP+FN\frac{TP}{TP+FN}
3. F1-score C. 2⋅P⋅RP+R2\cdot\frac{P\cdot R}{P+R}
4. Specificity D. TNTN+FP\frac{TN}{TN+FP}

Q12. Match each search/optimization method to its description.

Method Description
1. Grid search A. Uses a surrogate model to choose promising hyperparameters
2. Random search B. Exhaustively tries every combination in a defined grid
3. Bayesian optimization C. Samples hyperparameter combinations at random
4. Stacking D. Trains a meta-model on base learners' predictions

Section C — True/False WITH Justification (3 marks each: 1 T/F + 2 justification)

Q13. "Increasing model complexity always reduces test error." State True or False and justify.

Q14. "On an imbalanced dataset (99% negatives), a model predicting 'negative' always achieves 99% accuracy but 0% recall on the positive class." State True or False and justify.

Q15. "A large gap between low training error and high validation error indicates underfitting." State True or False and justify.

Q16. "Moving the classification threshold to increase recall generally decreases precision." State True or False and justify.

Q17. "RMSE is always greater than or equal to MAE for the same set of errors." State True or False and justify.

Answer keyMark scheme & solutions

Section A (1 mark each)

Q1. B — Underfitting. High bias = model too simple to capture patterns; low variance = stable across datasets → underfitting. (1)

Q2. B — 200. 1000/5=2001000/5 = 200 per fold. (1)

Q3. B. Precision = TP/(TP+FP)TP/(TP+FP) (fraction of predicted positives that are correct). Option A is recall. (1)

Q4. B — TPR vs FPR. ROC plots sensitivity against false positive rate across thresholds. (1)

Q5. C — k=nk=n. Each fold holds out exactly one sample. (1)

Q6. B — MSE. Squaring amplifies large errors relative to MAE. (1)

Q7. B — no better than random. AUC 0.5 = diagonal ROC = random guessing. (1)

Q8. B. Outer loop estimates generalization; inner loop tunes hyperparameters, avoiding optimistic bias. (1)

Q9. B. Stratification preserves class ratios in each fold. (1)

Q10. B — Confident but wrong. −log⁡(p)-\log(p) blows up as p→0p\to 0 for the true class. (1)

Q11. 1→B, 2→A, 3→C, 4→D. (2)

Q12. 1→B, 2→C, 3→A, 4→D. (2)

Section C (3 marks each)

Q13. FALSE. (T/F: 1) Increasing complexity reduces training error, but test error follows a U-shape: it decreases then rises again once the model overfits (variance dominates). So it does not always reduce test error. (justification: 2)

Q14. TRUE. (1) Predicting all negatives: TN=990, FN=10TN=990,\ FN=10 (per 1000), TP=0TP=0. Accuracy =990/1000=99%=990/1000=99\%. Recall =TP/(TP+FN)=0/10=0%=TP/(TP+FN)=0/10=0\%. Demonstrates accuracy is misleading under imbalance. (2)

Q15. FALSE. (1) Low training + high validation error = high variance = overfitting, not underfitting. Underfitting shows high error on both training and validation. (2)

Q16. TRUE. (1) Lowering the threshold labels more instances positive → catches more true positives (↑recall) but admits more false positives (↓precision). This is the precision–recall tradeoff. (2)

Q17. TRUE. (1) By the QM–AM inequality (or Jensen's for the convex square), RMSE=mean(e2)≥mean(∣e∣)=MAE\text{RMSE}=\sqrt{\text{mean}(e^2)} \ge \text{mean}(|e|)=\text{MAE}, with equality only when all ∣ei∣|e_i| are equal. (2)

[
  {"claim":"5-fold CV on 1000 samples gives 200 per fold","code":"result = (1000/5 == 200)"},
  {"claim":"All-negative model: accuracy 99%, recall 0% (990 TN,10 FN,0 TP)","code":"TN,FN,TP,FP=990,10,0,0; acc=(TP+TN)/(TP+TN+FP+FN); rec=TP/(TP+FN); result = (acc==0.99 and rec==0.0)"},
  {"claim":"RMSE >= MAE for errors [1,3,5]","code":"e=[1,3,5]; mae=sum(abs(x) for x in e)/3; rmse=(sum(x**2 for x in e)/3)**0.5; result = rmse>=mae"},
  {"claim":"AUC=0.5 corresponds to random classifier (equal to diagonal)","code":"result = (0.5 == 0.5)"}
]