Level 3 — ProductionLinear & Logistic Regression

Linear & Logistic Regression

⏱ 45 minutes60 marksprintable — key stays hidden on paper

Time limit: 45 minutes Total marks: 60 Instructions: Show all derivations from first principles. Where code is asked, pseudo-code or NumPy-style is acceptable but must be correct and runnable in spirit. Use $...$ for math.

Q1. Ordinary Least Squares — full derivation (12 marks)

For simple linear regression $\hat{y}_i = \beta_0 + \beta_1 x_i$ :

(a) Write the residual sum of squares cost $J(\beta_0,\beta_1)$ and derive the OLS estimates $\hat\beta_0,\hat\beta_1$ by setting the partial derivatives to zero. Show every algebraic step. (8)

(b) Show that the fitted line passes through $(\bar x, \bar y)$ . (2)

Q2. Normal equation & ridge (10 marks)

(a) Starting from the vectorised cost $J(\boldsymbol\beta)=\lVert \mathbf{y}-\mathbf{X}\boldsymbol\beta\rVert^2$ , derive the normal equation and give the closed-form solution $\hat{\boldsymbol\beta}$ . (4)

(b) Repeat for L2 (ridge) cost $J=\lVert \mathbf{y}-\mathbf{X}\boldsymbol\beta\rVert^2 + \lambda\lVert\boldsymbol\beta\rVert^2$ and give the ridge closed-form. (3)

(c) Explain, in one or two sentences, why the ridge solution is invertible even when $\mathbf{X}^\top\mathbf{X}$ is singular. (3)

Q3. Gradient descent from memory (10 marks)

(a) For MSE cost $J(\boldsymbol\beta)=\frac{1}{2m}\lVert\mathbf{X}\boldsymbol\beta-\mathbf{y}\rVert^2$ , derive the gradient $\nabla_{\boldsymbol\beta}J$ . (3)

(b) Write NumPy-style pseudocode for batch gradient descent that fits $\boldsymbol\beta$ (include the update rule, learning rate, and iteration loop). (5)

Q4. Logistic regression & log-loss (12 marks)

(a) Define the sigmoid $\sigma(z)$ and prove that $\sigma'(z)=\sigma(z)(1-\sigma(z))$ . (3)

(b) Write the binary cross-entropy (log-loss) for one example, then derive the gradient of the total log-loss w.r.t. the weights $\mathbf{w}$ , showing that it simplifies to $\frac{1}{m}\mathbf{X}^\top(\boldsymbol{\hat p}-\mathbf{y})$ . (6)

(c) Given weights $\mathbf{w}=[1,\,2]$ and bias $b=-3$ for features $\mathbf{x}=[1,\,1]$ , compute the predicted probability and state the predicted class at threshold 0.5. (3)

Q5. Decision boundary & softmax (10 marks)

(a) Show that the decision boundary of binary logistic regression at threshold 0.5 is the linear surface $\mathbf{w}^\top\mathbf{x}+b=0$ . (3)

(b) Write the softmax function for $K$ classes and state the predicted class rule. (3)

(c) For logits $\mathbf{z}=[2,\,1,\,0]$ compute the softmax probabilities (leave in exponential form then numeric to 3 dp) and give the predicted class. (4)

Q6. Model evaluation & regularization reasoning (6 marks)

(a) Define $R^2$ and adjusted $R^2$ ; explain why adjusted $R^2$ is preferred when comparing models with different numbers of predictors. (3)

(b) Contrast L1 (Lasso) and L2 (Ridge): which induces sparsity and why? One sentence on when Elastic Net is preferred. (3)

Answer keyMark scheme & solutions

Q1 (12)

(a) $J=\sum_i (y_i-\beta_0-\beta_1 x_i)^2$ . (1)

Partial derivatives: $\frac{\partial J}{\partial\beta_0}=-2\sum(y_i-\beta_0-\beta_1 x_i)=0 \Rightarrow \sum y_i = m\beta_0+\beta_1\sum x_i.$ (2) $\frac{\partial J}{\partial\beta_1}=-2\sum x_i(y_i-\beta_0-\beta_1 x_i)=0.$ (2) From the first equation: $\beta_0=\bar y-\beta_1\bar x$ . (1) Substituting and simplifying gives: $\hat\beta_1=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2},\qquad \hat\beta_0=\bar y-\hat\beta_1\bar x.$ (2)

(b) From $\hat\beta_0=\bar y-\hat\beta_1\bar x$ , at $x=\bar x$ : $\hat y=\hat\beta_0+\hat\beta_1\bar x=\bar y$ . Hence line passes through $(\bar x,\bar y)$ . (2)

(c) Any of: errors have zero mean, constant variance (homoscedasticity), are uncorrelated; predictors non-random / full rank (Gauss–Markov conditions). (2)

Q2 (10)

(a) $J=(\mathbf y-\mathbf X\boldsymbol\beta)^\top(\mathbf y-\mathbf X\boldsymbol\beta)$ . Gradient: $\nabla J=-2\mathbf X^\top(\mathbf y-\mathbf X\boldsymbol\beta)=0 \Rightarrow \mathbf X^\top\mathbf X\boldsymbol\beta=\mathbf X^\top\mathbf y.$ (2) $\boxed{\hat{\boldsymbol\beta}=(\mathbf X^\top\mathbf X)^{-1}\mathbf X^\top\mathbf y.}$ (2)

(b) $\nabla J=-2\mathbf X^\top(\mathbf y-\mathbf X\boldsymbol\beta)+2\lambda\boldsymbol\beta=0$ . (1) $\hat{\boldsymbol\beta}=(\mathbf X^\top\mathbf X+\lambda\mathbf I)^{-1}\mathbf X^\top\mathbf y.$ (2)

(c) $\mathbf X^\top\mathbf X$ is positive semi-definite; adding $\lambda\mathbf I$ ( $\lambda>0$ ) shifts all eigenvalues up by $\lambda>0$ , making the matrix positive-definite hence invertible. (3)

Q3 (10)

(a) $\nabla J=\frac{1}{m}\mathbf X^\top(\mathbf X\boldsymbol\beta-\mathbf y)$ . Derivation: differentiate $\frac{1}{2m}(\mathbf X\boldsymbol\beta-\mathbf y)^\top(\mathbf X\boldsymbol\beta-\mathbf y)$ . (3)

(b)

def gd(X, y, lr=0.01, n_iter=1000):
    m, n = X.shape
    beta = np.zeros(n)
    for _ in range(n_iter):
        grad = (1/m) * X.T @ (X @ beta - y)
        beta -= lr * grad
    return beta

(5: correct init 1, grad 2, update 1, loop 1)

(c) (i) Too large → overshoots minimum, cost may oscillate or diverge. (ii) Too small → convergence very slow, may need excessive iterations. (2)

Q4 (12)

(a) $\sigma(z)=\frac{1}{1+e^{-z}}$ . (1) $\sigma'(z)=\frac{e^{-z}}{(1+e^{-z})^2}=\sigma(z)\cdot\frac{e^{-z}}{1+e^{-z}}=\sigma(z)(1-\sigma(z))$ . (2)

(b) Per example: $L=-[y\log \hat p+(1-y)\log(1-\hat p)]$ with $\hat p=\sigma(\mathbf w^\top\mathbf x)$ . (2) $\frac{\partial L}{\partial \hat p}=-\frac{y}{\hat p}+\frac{1-y}{1-\hat p}$ ; chain with $\hat p'=\hat p(1-\hat p)\mathbf x$ gives $\frac{\partial L}{\partial\mathbf w}=(\hat p-y)\mathbf x$ . (3) Summing over $m$ examples and averaging: $\nabla J=\frac{1}{m}\mathbf X^\top(\boldsymbol{\hat p}-\mathbf y)$ . (1)

(c) $z=1\cdot1+2\cdot1-3=0$ , $\sigma(0)=0.5$ . At threshold 0.5, class = 1 (or boundary case; either accepted with justification). (3)

Q5 (10)

(a) Predict class 1 when $\hat p\ge0.5\Leftrightarrow\sigma(\mathbf w^\top\mathbf x+b)\ge0.5\Leftrightarrow \mathbf w^\top\mathbf x+b\ge0$ . Boundary is $\mathbf w^\top\mathbf x+b=0$ , linear in $\mathbf x$ . (3)

(b) $\text{softmax}(\mathbf z)_k=\frac{e^{z_k}}{\sum_j e^{z_j}}$ ; predict $\arg\max_k z_k$ . (3)

(c) $e^2=7.389,\ e^1=2.718,\ e^0=1$ ; sum $=11.107$ . $p=[0.665,\,0.245,\,0.090]$ . Predicted class = 0 (first). (4)

Q6 (6)

(a) $R^2=1-\frac{SS_{res}}{SS_{tot}}$ . $R^2_{adj}=1-\frac{(1-R^2)(m-1)}{m-p-1}$ . Adjusted $R^2$ penalizes adding predictors, so it doesn't rise spuriously from extra features—fairer for model comparison. (3)

(b) L1 induces sparsity: its diamond-shaped constraint has corners on axes so coefficients hit exactly zero; L2 shrinks but rarely zeroes. Elastic Net preferred with many correlated features (groups selection + stability). (3)

[
{"claim":"sigmoid(0)=0.5","code":"z=0; result=(1/(1+exp(-z))==Rational(1,2))"},
{"claim":"softmax of [2,1,0] first prob approx 0.665","code":"import math; z=[2,1,0]; e=[math.exp(v) for v in z]; s=sum(e); result=abs(e[0]/s-0.665)<0.01"},
{"claim":"ridge shifts eigenvalues by lambda making invertible","code":"lam=2; M=Matrix([[0,0],[0,0]])+lam*eye(2); result=(M.det()!=0)"},
{"claim":"OLS beta1 for x=[1,2,3],y=[2,4,6] equals 2","code":"import numpy as np; x=np.array([1,2,3]); y=np.array([2,4,6]); b1=np.sum((x-x.mean())*(y-y.mean()))/np.sum((x-x.mean())**2); result=abs(b1-2)<1e-9"}
]