Level 4 — ApplicationLinear & Logistic Regression

Linear & Logistic Regression

⏱ 60 minutes60 marksprintable — key stays hidden on paper

Level 4 — Application (novel problems, no hints) Time limit: 60 minutes Total marks: 60

Answer all questions. Show all working. Use $...$ for inline math and $...$ for displayed equations.

Question 1 — OLS from scratch on a tiny dataset (12 marks)

You are given the following four data points for a simple linear regression $y = \beta_0 + \beta_1 x$ :

$x$	1	2	3	4
$y$	2	2	4	5

(a) Using the OLS closed-form estimators, compute $\hat\beta_1$ and $\hat\beta_0$ . (5)

(b) Compute the fitted values, residuals, and the residual sum of squares (RSS). (4)

Question 2 — Ridge vs Normal equation (14 marks)

A multiple linear regression uses the design matrix $X$ (with a bias column) and target $y$ :

$X = \begin{bmatrix} 1 & 0 \\ 1 & 2 \\ 1 & 4 \end{bmatrix}, \qquad y = \begin{bmatrix} 1 \\ 2 \\ 5 \end{bmatrix}$

(a) State the normal equation and compute the OLS coefficient vector $\hat\beta = (X^\top X)^{-1}X^\top y$ . (6)

(b) Now apply L2 (Ridge) regularization with $\lambda = 2$ , using $\hat\beta_{ridge} = (X^\top X + \lambda I)^{-1}X^\top y$ (regularizing all coefficients including bias, for simplicity). Compute the new coefficient vector. (5)

Question 3 — Logistic regression decision boundary & log-loss (14 marks)

A trained binary logistic regression classifier has parameters $w_0 = -3$ , $w_1 = 1$ , $w_2 = 2$ for the model $\hat p = \sigma(w_0 + w_1 x_1 + w_2 x_2), \qquad \sigma(z) = \frac{1}{1+e^{-z}}.$

(a) Derive the equation of the decision boundary (where $\hat p = 0.5$ ) in the $(x_1, x_2)$ plane and state its slope and intercept. (4)

(b) For the point $(x_1, x_2) = (1, 2)$ with true label $y = 1$ , compute $\hat p$ and the log-loss (binary cross-entropy) contribution of this single point. Use $\ln$ . (6)

(c) A colleague claims "logistic regression is a linear model." Justify or refute this in terms of what is linear and what is not. (4)

Question 4 — Polynomial regression, overfitting & regularization choice (10 marks)

You fit polynomial regression models of degree $d = 1, 3, 9$ to 12 noisy training points.

(a) Sketch (describe qualitatively) how training error and validation error typically behave as $d$ increases from 1 to 9, and identify the regime of underfitting vs overfitting. (4)

(b) You want a degree-9 model but must control overfitting AND perform automatic feature selection (drive some coefficients exactly to zero). Which regularization method do you choose and why? Give the objective function you would minimize. (4)

(c) Adjusted $R^2$ for a fit is defined as $\bar R^2 = 1 - (1-R^2)\frac{n-1}{n-p-1}$ . For $n = 12$ , $p = 9$ , and $R^2 = 0.95$ , compute $\bar R^2$ and comment on what the gap between $R^2$ and $\bar R^2$ signals. (2)

Question 5 — Gradient descent update & assumptions (10 marks)

Consider MSE cost $J(\beta) = \frac{1}{2m}\sum_{i=1}^m (\beta_0 + \beta_1 x_i - y_i)^2$ .

(a) Derive $\partial J/\partial \beta_0$ and $\partial J/\partial \beta_1$ and write the gradient descent update rules with learning rate $\alpha$ . (4)

(b) Using the dataset from Question 1, starting from $\beta_0 = 0, \beta_1 = 0$ and $\alpha = 0.1$ , compute the coefficient values after one gradient descent step. (4)

(c) Name two core assumptions of linear regression that, if violated, would make the OLS standard errors unreliable. (2)

Answer keyMark scheme & solutions

Question 1 (12 marks)

(a) Means: $\bar x = 2.5$ , $\bar y = 3.25$ . $S_{xy} = \sum (x_i-\bar x)(y_i-\bar y)$ : deviations $x$ : $(-1.5,-0.5,0.5,1.5)$ , $y$ : $(-1.25,-1.25,0.75,1.75)$ . Products: $1.875 + 0.625 + 0.375 + 2.625 = 5.5$ . $S_{xx} = \sum(x_i-\bar x)^2 = 2.25+0.25+0.25+2.25 = 5$ . $\hat\beta_1 = 5.5/5 = 1.1$ (3) $\hat\beta_0 = \bar y - \hat\beta_1\bar x = 3.25 - 1.1(2.5) = 3.25 - 2.75 = 0.5$ (2)

(b) Fitted $\hat y = 0.5 + 1.1x$ : at $x=1,2,3,4$ → $1.6, 2.7, 3.8, 4.9$ . (1) Residuals $y-\hat y$ : $0.4, -0.7, 0.2, 0.1$ . (1) RSS $= 0.16 + 0.49 + 0.04 + 0.01 = 0.70$ . (2)

(c) $SS_{tot} = \sum(y_i-\bar y)^2 = 1.5625+1.5625+0.5625+3.0625 = 6.75$ . $R^2 = 1 - 0.70/6.75 = 1 - 0.10370 = 0.8963$ . (2) Interpretation: ~89.6% of the variance in $y$ is explained by the linear model. (1)

Question 2 (14 marks)

(a) Normal equation: $X^\top X\,\hat\beta = X^\top y$ . (1) $X^\top X = \begin{bmatrix} 3 & 6 \\ 6 & 20 \end{bmatrix}$ , $X^\top y = \begin{bmatrix} 8 \\ 24 \end{bmatrix}$ (since $\sum y=8$ , $\sum x y = 0+4+20=24$ ). (2) $\det = 60-36 = 24$ . Inverse $= \frac{1}{24}\begin{bmatrix} 20 & -6 \\ -6 & 3 \end{bmatrix}$ . $\hat\beta = \frac{1}{24}\begin{bmatrix} 20\cdot8 - 6\cdot24 \\ -6\cdot8 + 3\cdot24 \end{bmatrix} = \frac{1}{24}\begin{bmatrix} 16 \\ 24 \end{bmatrix} = \begin{bmatrix} 0.6667 \\ 1 \end{bmatrix}$ . (3) So $\hat\beta_0 = 2/3$ , $\hat\beta_1 = 1$ .

(b) $X^\top X + 2I = \begin{bmatrix} 5 & 6 \\ 6 & 22 \end{bmatrix}$ , $\det = 110-36 = 74$ . Inverse $= \frac{1}{74}\begin{bmatrix} 22 & -6 \\ -6 & 5 \end{bmatrix}$ . $\hat\beta_{ridge} = \frac{1}{74}\begin{bmatrix} 22\cdot8 - 6\cdot24 \\ -6\cdot8 + 5\cdot24 \end{bmatrix} = \frac{1}{74}\begin{bmatrix} 32 \\ 72 \end{bmatrix} = \begin{bmatrix} 0.4324 \\ 0.9730 \end{bmatrix}$ . (5) (Both coefficients shrunk relative to OLS.)

(c) Ridge penalty $\lambda\sum\beta_j^2$ has gradient $2\lambda\beta_j$ , which vanishes as $\beta_j\to0$ , so shrinkage weakens near zero and the optimum sits at a small nonzero value; the smooth quadratic penalty never produces a "corner" at zero. Lasso's $|\beta_j|$ has a constant-magnitude subgradient that can pin coefficients exactly at zero. (3)

Question 3 (14 marks)

(a) $\hat p = 0.5 \Rightarrow z = 0 \Rightarrow -3 + x_1 + 2x_2 = 0$ . (2) Solve for $x_2$ : $x_2 = \frac{3 - x_1}{2} = 1.5 - 0.5x_1$ . Slope $= -0.5$ , intercept $= 1.5$ . (2)

(b) $z = -3 + 1(1) + 2(2) = 2$ . (2) $\hat p = \sigma(2) = \frac{1}{1+e^{-2}} = \frac{1}{1+0.13534} = 0.8808$ . (2) Log-loss for $y=1$ : $-\ln \hat p = -\ln(0.8808) = 0.1269$ . (2)

(c) The model is linear in the parameters/inputs inside $z = w^\top x$ — the decision boundary is a hyperplane. But the output probability is a nonlinear function of $z$ (the sigmoid). So it is a generalized linear model: linear in the log-odds ( $\ln\frac{p}{1-p} = w^\top x$ ) but nonlinear in probability space. The colleague is right about the boundary/log-odds but the mapping to probability is nonlinear. (4)

Question 4 (10 marks)

(a) Training error decreases monotonically as $d$ rises (more flexibility fits noise). Validation error is U-shaped: high at $d=1$ (underfitting / high bias), minimum at some moderate $d$ , then rises at $d=9$ (overfitting / high variance). $d=1$ underfits, $d=9$ overfits. (4)

(b) Choose L1 (Lasso) — its $\ell_1$ penalty drives some coefficients exactly to zero, giving automatic feature selection while controlling variance. (Elastic Net also acceptable if selection + grouping desired.) (2) Objective: $\displaystyle \min_\beta \frac{1}{2m}\sum_{i=1}^m (\hat y_i - y_i)^2 + \lambda\sum_{j=1}^{9}|\beta_j|$ . (2)

(c) $\bar R^2 = 1 - (1-0.95)\frac{12-1}{12-9-1} = 1 - 0.05\cdot\frac{11}{2} = 1 - 0.275 = 0.725$ . (1) The large drop (0.95 → 0.725) signals the high $R^2$ is inflated by using many parameters ( $p=9$ ) relative to few observations ( $n=12$ ) — a sign of overfitting. (1)

Question 5 (10 marks)

(a) $\frac{\partial J}{\partial\beta_0} = \frac{1}{m}\sum(\hat y_i - y_i)$ , $\frac{\partial J}{\partial\beta_1} = \frac{1}{m}\sum(\hat y_i - y_i)x_i$ . (2) Updates: $\beta_0 \leftarrow \beta_0 - \alpha\frac{1}{m}\sum(\hat y_i-y_i)$ , $\beta_1 \leftarrow \beta_1 - \alpha\frac{1}{m}\sum(\hat y_i-y_i)x_i$ . (2)

(b) With $\beta=0$ , $\hat y_i = 0$ , so errors $= -y_i = (-2,-2,-4,-5)$ , $m=4$ . $\frac{\partial J}{\partial\beta_0} = \frac{1}{4}(-2-2-4-5) = -13/4 = -3.25$ . $\frac{\partial J}{\partial\beta_1} = \frac{1}{4}(-2\cdot1 -2\cdot2 -4\cdot3 -5\cdot4) = \frac{1}{4}(-2-4-12-20)=-38/4 = -9.5$ . (2) $\beta_0 = 0 - 0.1(-3.25) = 0.325$ ; $\beta_1 = 0 - 0.1(-9.5) = 0.95$ . (2)

(c) Any two: (i) homoscedasticity (constant error variance); (ii) independence of errors (no autocorrelation); (iii) no perfect multicollinearity; (iv) normality of errors (for exact small-sample inference). (2)

[
  {"claim":"Q1 OLS slope=1.1, intercept=0.5, RSS=0.70, R2≈0.8963","code":"x=Matrix([1,2,3,4]); y=Matrix([2,2,4,5]); xb=Rational(sum(x),4); yb=Rational(sum(y),4); Sxy=sum((x[i]-xb)*(y[i]-yb) for i in range(4)); Sxx=sum((x[i]-xb)**2 for i in range(4)); b1=Sxy/Sxx; b0=yb-b1*xb; fit=[b0+b1*x[i] for i in range(4)]; RSS=sum((y[i]-fit[i])**2 for i in range(4)); SST=sum((y[i]-yb)**2 for i in range(4)); R2=1-RSS/SST; result=(b1==Rational(11,10)) and (b0==Rational(1,2)) and (RSS==Rational(7,10)) and (abs(float(R2)-0.8963)<1e-3)"},
  {"claim":"Q2 OLS beta=[2/3,1], ridge(lambda2)=[16/37,36/37]","code":"X=Matrix([[1,0],[1,2],[1,4]]); y=Matrix([1,2,5]); XtX=X.T*X; Xty=X.T*y; b=XtX.inv()*Xty; br=(XtX+2*eye(2)).inv()*Xty; result=(b==Matrix([Rational(2,3),1])) and (br==Matrix([Rational(16,37),Rational(36,37)]))"},
  {"claim":"Q3 phat=sigma(2)=0.8808, logloss=0.1269","code":"import math; p=1/(1+math.exp(-2)); ll=-math.log(p); result=(abs(p-0.8808)<1e-3) and (abs(ll-0.1269)<1e-3)"},
  {"claim":"Q4 adjusted R2=0.725","code":"n=12;pp=9;R2=Rational(95,100); adj=1-(1-R2)*Rational(n-1,n-pp-1); result=adj==Rational(725,1000)"},
  {"claim":"Q5 one GD step gives beta0=0.325, beta1=0.95","code":"x=[1,2,3,4]; y=[2,2,4,5]; m=4; a=Rational(1,10); err=[0-yi for yi in y]; g0=Rational(sum(err),m); g1=Rational(sum(err[i]*x[i] for i in range(4)),m); b0=0-a*g0; b1=0-a*g1; result=(b0==Rational(325,1000)) and (b1==Rational(95,100))"}
]