Level 5 — MasteryData Preprocessing & Feature Engineering

Data Preprocessing & Feature Engineering

⏱ 90 minutes60 marksprintable — key stays hidden on paper

Level 5 — Mastery (cross-domain: math + statistics + coding + proof) Time limit: 90 minutes Total marks: 60

Instructions: Answer all three questions. Show derivations, justify every modeling choice, and state assumptions. Code may be written in Python (NumPy/pandas/scikit-learn style pseudocode acceptable if correct).

Question 1 — Scaling, Leakage, and the Pipeline Contract (20 marks)

A dataset has a single feature $x$ . You split into train and test sets and apply z-score standardization.

(a) Prove that after standardization using the training statistics $\mu_{tr}, \sigma_{tr}$ , the transformed training feature has sample mean $0$ and (population) variance $1$ . Show the algebra. (4 marks)

(b) A colleague computes $\mu, \sigma$ over the entire dataset (train + test combined) before splitting. Explain precisely why this constitutes data leakage, and quantify it: given training values $\{2,4,6,8\}$ and test values $\{100, 102\}$ , compute the train feature's standardized mean under the correct (train-only) procedure versus the leaky (full-data) procedure. (6 marks)

(c) Min-max scaling maps $x \mapsto \frac{x - x_{min}}{x_{max} - x_{min}}$ . Prove that min-max scaling and z-score standardization are related by an affine transformation on any fixed dataset, i.e. find constants $a, b$ such that $z_i = a\,m_i + b$ where $m_i$ is the min-max value and $z_i$ the z-score of point $i$ . Express $a, b$ in terms of range $R = x_{max}-x_{min}$ , $\sigma$ , and $\mu, x_{min}$ . (6 marks)

(d) State one concrete scenario where min-max scaling is preferable to standardization, and one where it is dangerous. (4 marks)

Question 2 — Outliers, Transformations, and Distributional Reasoning (20 marks)

Consider positive-valued feature data $x > 0$ that is right-skewed.

(a) For the sample $\{1, 2, 2, 3, 3, 3, 4, 5, 8, 27\}$ , compute $Q_1$ , $Q_3$ , the IQR, and the Tukey upper fence $Q_3 + 1.5\cdot\text{IQR}$ . Identify which points are flagged as outliers. Use linear interpolation quartiles (positions $0.25(n+1)$ and $0.75(n+1)$ ). (6 marks)

(b) Apply a $\log$ transform $y = \ln(x)$ to the value $27$ and to the median. Argue mathematically (using the derivative of $\ln$ ) why log transforms compress large values more than small values, and hence reduce right-skew and outlier influence. (6 marks)

(c) The Box–Cox transform is $x^{(\lambda)} = \frac{x^\lambda - 1}{\lambda}$ for $\lambda \neq 0$ . Prove that $\lim_{\lambda \to 0} \frac{x^\lambda - 1}{\lambda} = \ln x$ , justifying why $\ln$ is the continuous $\lambda=0$ case. (4 marks)

(d) Write correct pseudocode that: fits the outlier fences and log-transform parameters on the training set only, then applies them to a held-out set — and explain the leakage risk if done otherwise. (4 marks)

Question 3 — Encoding, Imbalance, and Correlation (20 marks)

A classification dataset has a categorical feature city (4 levels), a binary target y, and class imbalance (positive rate 5%).

(a) Contrast one-hot encoding vs target (mean) encoding for city. Derive the target-encoded value for a city with 3 samples having targets $\{1,0,1\}$ using additive (Laplace) smoothing with prior = global mean $\bar y = 0.05$ and smoothing weight $m=10$ : $\hat{e} = \frac{n\bar{y}_c + m\bar{y}}{n+m}$ . (6 marks)

(b) Explain the leakage mechanism in naive target encoding and how out-of-fold / smoothed encoding mitigates it. (4 marks)

(c) SMOTE generates a synthetic minority point as $x_{new} = x_i + \delta(x_{nn} - x_i)$ , $\delta \sim U(0,1)$ . Given minority points $x_i=(1,2)$ and nearest neighbor $x_{nn}=(3,6)$ with $\delta=0.25$ , compute $x_{new}$ . Prove that every SMOTE point lies on the segment between the two points and state one risk of SMOTE with respect to outliers. (6 marks)

(d) Two features have Pearson correlation $r = 0.98$ . Compute the Variance Inflation Factor $\text{VIF} = \frac{1}{1-R^2}$ (with $R^2 = r^2$ for the two-feature case) and interpret whether multicollinearity is a concern. (4 marks)

Answer keyMark scheme & solutions

Question 1

(a) [4 marks] Standardized value $z_i = \frac{x_i - \mu_{tr}}{\sigma_{tr}}$ . Mean: $\bar z = \frac{1}{n}\sum \frac{x_i-\mu_{tr}}{\sigma_{tr}} = \frac{1}{\sigma_{tr}}\left(\frac{1}{n}\sum x_i - \mu_{tr}\right) = \frac{1}{\sigma_{tr}}(\mu_{tr}-\mu_{tr}) = 0$ . (2) Variance: $\frac{1}{n}\sum (z_i-0)^2 = \frac{1}{n}\sum \frac{(x_i-\mu_{tr})^2}{\sigma_{tr}^2} = \frac{\sigma_{tr}^2}{\sigma_{tr}^2}=1$ . (2)

(b) [6 marks] Leakage: test-set information (its mean/spread) influences the training transform; the model implicitly "sees" future data, inflating optimism and breaking the independence of the test set. (2)

Correct (train-only): $\mu_{tr}=\frac{2+4+6+8}{4}=5$ , $\sigma_{tr}^2=\frac{9+1+1+9}{4}=5$ , $\sigma_{tr}=\sqrt5\approx2.236$ . Standardized train mean $=0$ by part (a). (2)

Leaky (full data, 6 points): $\mu_{full}=\frac{2+4+6+8+100+102}{6}=\frac{222}{6}=37$ . Train standardized mean $=\frac{1}{4}\sum\frac{x_i-37}{\sigma_{full}}$ . Train raw mean $=5$ , so numerator mean $=5-37=-32$ ; $\sigma_{full}=\sqrt{\text{Var}_{full}}$ . Var $_{full}=\frac{1}{6}\sum(x_i-37)^2 = \frac{1225+1089+961+841+3969+4225}{6}=\frac{12310}{6}\approx2051.67$ , $\sigma_{full}\approx45.30$ . Leaky train mean $=\frac{-32}{45.30}\approx-0.706$ . (2) → Correct gives $0$ ; leaky gives $\approx -0.71$ , showing distortion.

(c) [6 marks] $m_i=\frac{x_i-x_{min}}{R}$ , $z_i=\frac{x_i-\mu}{\sigma}$ . Solve $x_i = m_i R + x_{min}$ ; substitute: $z_i=\frac{m_i R + x_{min}-\mu}{\sigma}=\frac{R}{\sigma}m_i + \frac{x_{min}-\mu}{\sigma}$ . So $a=\dfrac{R}{\sigma}$ , $b=\dfrac{x_{min}-\mu}{\sigma}$ — an affine map. (6) (4 for form, 2 for correct constants)

(d) [4 marks] Preferable: bounded-input requirements, e.g. image pixel intensities to $[0,1]$ , or neural nets with sigmoid activations / distance metrics needing fixed range. (2) Dangerous: presence of outliers — a single extreme $x_{max}$ compresses all other values into a tiny sub-interval, destroying resolution; standardization (or robust scaling) is safer. (2)

Question 2

(a) [6 marks] Sorted: $1,2,2,3,3,3,4,5,8,27$ , $n=10$ . $Q_1$ position $=0.25(11)=2.75$ → between 2nd(2) and 3rd(2): $Q_1=2+0.75(0)=2$ . (2) $Q_3$ position $=0.75(11)=8.25$ → between 8th(5) and 9th(8): $Q_3=5+0.25(3)=5.75$ . (2) IQR $=5.75-2=3.75$ ; upper fence $=5.75+1.5(3.75)=5.75+5.625=11.375$ . (1) Outlier: $27>11.375$ ⇒ flagged (only 27). (1)

(b) [6 marks] $\ln 27 \approx 3.296$ ; median (avg of 5th,6th $=3,3$ ) $=3$ , $\ln 3\approx1.099$ . (2) Raw gap 27 vs 3 = 24; log gap $\approx 2.197$ — hugely compressed. (1) Since $\frac{d}{dx}\ln x = \frac{1}{x}$ , the transform's slope decreases as $x$ grows; large $x$ get mapped with smaller local stretch, so wide right-tail spacing collapses, reducing skew and outlier leverage. (3)

(c) [4 marks] $\lim_{\lambda\to0}\frac{x^\lambda-1}{\lambda}$ : indeterminate $0/0$ . Write $x^\lambda=e^{\lambda\ln x}$ . Either L'Hôpital w.r.t. $\lambda$ : $\frac{d}{d\lambda}(x^\lambda-1)=x^\lambda\ln x$ , denominator derivative $=1$ , limit $=x^0\ln x=\ln x$ . (4) (2 for method, 2 for result)

(d) [4 marks]

# TRAIN ONLY
q1, q3 = np.percentile(X_train, [25,75])
iqr = q3 - q1
upper = q3 + 1.5*iqr          # fences fit on train
# choose lambda / use log; store shift if needed
X_train_t = np.log(X_train + eps)
 
# APPLY to test using SAME fences/transform
X_test_clipped = np.clip(X_test, None, upper)
X_test_t = np.log(X_test_clipped + eps)

Leakage risk: fitting fences/transform params on test (or full) data lets test distribution influence preprocessing, biasing evaluation optimistically. (2 code, 2 explanation)

Question 3

(a) [6 marks] One-hot: 4 binary columns, no ordinal assumption, no target leakage, but high dimensionality/sparsity for many levels. Target encoding: single column = per-category target mean, compact and captures signal, but risks leakage/overfit on rare levels. (2) Smoothed encode: $n=3$ , $\bar y_c=\frac{1+0+1}{3}=\frac23\approx0.667$ , $\bar y=0.05$ , $m=10$ . $\hat e=\frac{3(0.667)+10(0.05)}{3+10}=\frac{2.0+0.5}{13}=\frac{2.5}{13}\approx0.1923$ . (4)

(b) [4 marks] Naive encoding uses the target of a row to build that row's feature → direct target leakage inflating training performance and overfitting rare categories. Out-of-fold encoding (compute a fold's encoding from other folds) and smoothing toward the global prior break the self-reference, shrinking unreliable small- $n$ estimates. (4)

(c) [6 marks] $x_{new}=x_i+\delta(x_{nn}-x_i)=(1,2)+0.25((3,6)-(1,2))=(1,2)+0.25(2,4)=(1+0.5,\,2+1)=(1.5,\,3)$ . (3) Proof: $x_{new}=(1-\delta)x_i+\delta x_{nn}$ is a convex combination with $\delta\in[0,1]$ , hence lies on the line segment between the two points. (2) Risk: if $x_i$ or $x_{nn}$ is an outlier/noisy minority point, SMOTE interpolates into unrealistic regions, amplifying noise and blurring class boundaries. (1)

(d) [4 marks] $R^2=r^2=0.98^2=0.9604$ ; $\text{VIF}=\frac{1}{1-0.9604}=\frac{1}{0.0396}\approx25.25$ . (2) VIF $\gg 10$ ⇒ severe multicollinearity; the two features are nearly redundant → unstable coefficient estimates; drop one or combine (PCA/regularization). (2)

[
  {"claim":"Leaky train standardized mean approx -0.706","code":"xs=[2,4,6,8,100,102]; mu=sum(xs)/len(xs); var=sum((x-mu)**2 for x in xs)/len(xs); sig=sqrt(var); train=[2,4,6,8]; m=sum((x-mu)/sig for x in train)/len(train); result=abs(float(m)-(-0.706))<0.01"},
  {"claim":"Tukey upper fence = 11.375 and 27 is the only outlier","code":"Q1=2; Q3=Rational(575,100); iqr=Q3-Q1; fence=Q3+Rational(3,2)*iqr; data=[1,2,2,3,3,3,4,5,8,27]; outs=[x for x in data if x>fence]; result=(fence==Rational(11375,1000)) and outs==[27]"},
  {"claim":"Box-Cox limit equals ln x","code":"lam,x=symbols('lam x',positive=True); result=limit((x**lam-1)/lam,lam,0)==log(x)"},
  {"claim":"Smoothed target encoding approx 0.1923","code":"e=(3*Rational(2,3)+10*Rational(5,100))/(3+10); result=abs(float(e)-0.1923)<0.001"},
  {"claim":"VIF approx 25.25 for r=0.98","code":"r=Rational(98,100); R2=r**2; vif=1/(1-R2); result=abs(float(vif)-25.2525)<0.01"},
  {"claim":"SMOTE point is (1.5,3)","code":"xi=Matrix([1,2]); xnn=Matrix([3,6]); d=Rational(1,4); xnew=xi+d*(xnn-xi); result=xnew==Matrix([Rational(3,2),3])"}
]