Noesis
Level 5 — MasteryLinear & Logistic Regression

Linear & Logistic Regression

⏱ 90 minutes60 marksprintable — key stays hidden on paper

Time limit: 90 minutes
Total marks: 60
Instructions: Answer all three questions. Show all derivations. Partial credit for correct method. Use ...... for math.

Question 1 — OLS, Ridge, and the Geometry of Least Squares (22 marks)

Consider the linear model y=Xβ+ε\mathbf{y} = X\boldsymbol{\beta} + \boldsymbol{\varepsilon} where X∈Rn×pX \in \mathbb{R}^{n\times p} (full column rank), y∈Rn\mathbf{y}\in\mathbb{R}^n.

(a) Starting from the cost J(β)=12∥y−Xβ∥22J(\boldsymbol\beta) = \tfrac{1}{2}\lVert \mathbf y - X\boldsymbol\beta\rVert_2^2, derive the normal equation and the closed-form OLS estimator β^\hat{\boldsymbol\beta}. Show explicitly that the stationary point is a minimum. (6)

(b) Prove that the OLS residual vector r=y−Xβ^\mathbf r = \mathbf y - X\hat{\boldsymbol\beta} is orthogonal to the column space of XX, and interpret y^=Xβ^\hat{\mathbf y} = X\hat{\boldsymbol\beta} geometrically as a projection. Give the projection ("hat") matrix HH and prove H2=HH^2 = H and H⊤=HH^\top = H. (6)

(c) For Ridge regression the cost becomes Jλ(β)=12∥y−Xβ∥22+λ2∥β∥22J_\lambda(\boldsymbol\beta)=\tfrac12\lVert \mathbf y - X\boldsymbol\beta\rVert_2^2 + \tfrac{\lambda}{2}\lVert\boldsymbol\beta\rVert_2^2, λ>0\lambda>0. Derive β^ridge\hat{\boldsymbol\beta}_{\text{ridge}} and explain why it is always defined even when X⊤XX^\top X is singular. (4)

(d) Given the tiny dataset (x,y)={(0,1),(1,3),(2,7)}(x,y)=\{(0,1),(1,3),(2,7)\} fit y=β0+β1xy=\beta_0+\beta_1 x by OLS. Compute β^0,β^1\hat\beta_0,\hat\beta_1, the fitted values, residuals, and R2R^2. (6)

Question 2 — Logistic Regression: Sigmoid, Log-Loss, Gradient (20 marks)

(a) Define the sigmoid σ(z)=11+e−z\sigma(z)=\dfrac{1}{1+e^{-z}}. Prove σ′(z)=σ(z)(1−σ(z))\sigma'(z)=\sigma(z)(1-\sigma(z)) and that σ(−z)=1−σ(z)\sigma(-z)=1-\sigma(z). (4)

(b) For binary labels yi∈{0,1}y_i\in\{0,1\} with p^i=σ(w⊤xi)\hat p_i=\sigma(\mathbf w^\top \mathbf x_i), write the binary cross-entropy (log-loss) L(w)L(\mathbf w) for nn samples. Derive ∇wL\nabla_{\mathbf w} L and show it equals ∑i(p^i−yi)xi\sum_i (\hat p_i - y_i)\mathbf x_i. Comment on why this gradient is identical in form to linear regression's. (8)

(c) A trained model has w=(w0,w1,w2)=(−3, 2, 1)\mathbf w=(w_0,w_1,w_2)=(-3,\,2,\,1) (with x0=1x_0=1 the bias). (i) Write the equation of the decision boundary in the (x1,x2)(x_1,x_2) plane and state its slope. (ii) For the point (x1,x2)=(1,2)(x_1,x_2)=(1,2) compute p^\hat p and the predicted class at threshold 0.50.5. (4)

(d) Interpret the coefficient w1=2w_1=2 in terms of the odds of the positive class when x1x_1 increases by one unit (holding x2x_2 fixed). Give the numerical odds multiplier. (4)

Question 3 — Regularization, Polynomial Fitting & Assumptions (18 marks)

(a) Write the Elastic Net objective combining L1 and L2 penalties. Explain, using the geometry of the constraint regions, why L1 (Lasso) produces exactly-zero coefficients (sparsity) while L2 (Ridge) does not. (6)

(b) You fit a degree-9 polynomial to 12 noisy points and get training R2=0.999R^2=0.999 but poor test performance. (i) Diagnose the phenomenon and relate it to the bias–variance trade-off. (ii) Give two concrete remedies and explain the mechanism of each. (5)

(c) A model reports R2=0.82R^2=0.82 with n=50n=50 observations and p=8p=8 predictors. Compute the adjusted R2R^2. Explain why adjusted R2R^2 is preferred when comparing models with different numbers of predictors. (4)

(d) State any three of the classical assumptions of linear regression and, for each, name one diagnostic or consequence of its violation. (3)

Answer keyMark scheme & solutions

Question 1

(a) [6] J=12(y−Xβ)⊤(y−Xβ)J=\tfrac12(\mathbf y-X\boldsymbol\beta)^\top(\mathbf y-X\boldsymbol\beta). Expand and differentiate: ∇βJ=−X⊤(y−Xβ)=0  ⇒  X⊤Xβ=X⊤y.\nabla_{\boldsymbol\beta}J = -X^\top(\mathbf y - X\boldsymbol\beta) = 0 \;\Rightarrow\; X^\top X\boldsymbol\beta = X^\top \mathbf y. (2 for gradient, 1 for normal eqn) Since XX is full column rank, X⊤XX^\top X is invertible: β^=(X⊤X)−1X⊤y\hat{\boldsymbol\beta}=(X^\top X)^{-1}X^\top\mathbf y. (2) Hessian ∇2J=X⊤X\nabla^2 J = X^\top X is positive definite (full rank) ⇒\Rightarrow strict minimum. (1)

(b) [6] Normal equation gives X⊤(y−Xβ^)=X⊤r=0X^\top(\mathbf y - X\hat{\boldsymbol\beta})=X^\top\mathbf r=0, so r⊥Col(X)\mathbf r \perp \text{Col}(X). (2) Thus y^\hat{\mathbf y} is the orthogonal projection of y\mathbf y onto Col(X)\text{Col}(X). (1) H=X(X⊤X)−1X⊤H=X(X^\top X)^{-1}X^\top. (1) H2=X(X⊤X)−1X⊤X(X⊤X)−1X⊤=X(X⊤X)−1X⊤=HH^2 = X(X^\top X)^{-1}X^\top X(X^\top X)^{-1}X^\top = X(X^\top X)^{-1}X^\top = H (idempotent). (1) H⊤=(X⊤)⊤((X⊤X)−1)⊤X⊤=X(X⊤X)−1X⊤=HH^\top = (X^\top)^\top((X^\top X)^{-1})^\top X^\top = X(X^\top X)^{-1}X^\top=H (symmetric, since X⊤XX^\top X symmetric). (1)

(c) [4] ∇Jλ=−X⊤(y−Xβ)+λβ=0⇒(X⊤X+λI)β=X⊤y\nabla J_\lambda = -X^\top(\mathbf y-X\boldsymbol\beta)+\lambda\boldsymbol\beta=0 \Rightarrow (X^\top X+\lambda I)\boldsymbol\beta = X^\top\mathbf y. (2) β^ridge=(X⊤X+λI)−1X⊤y\hat{\boldsymbol\beta}_{\text{ridge}}=(X^\top X+\lambda I)^{-1}X^\top\mathbf y. (1) X⊤XX^\top X is positive semidefinite (eigenvalues ≥0\ge0); adding λI\lambda I shifts all eigenvalues to ≥λ>0\ge\lambda>0, so the matrix is invertible even if X⊤XX^\top X is singular. (1)

(d) [6] xˉ=1, yˉ=1+3+73=113\bar x=1,\ \bar y=\tfrac{1+3+7}{3}=\tfrac{11}{3}. Sxx=∑(x−xˉ)2=1+0+1=2S_{xx}=\sum(x-\bar x)^2=1+0+1=2. Sxy=∑(x−xˉ)(y−yˉ)=(−1)(1−113)+0+(1)(7−113)=(−1)(−83)+(1)(103)=83+103=6S_{xy}=\sum(x-\bar x)(y-\bar y)=(-1)(1-\tfrac{11}{3})+0+(1)(7-\tfrac{11}{3})=(-1)(-\tfrac83)+(1)(\tfrac{10}{3})=\tfrac{8}{3}+\tfrac{10}{3}=6. β^1=Sxy/Sxx=6/2=3\hat\beta_1=S_{xy}/S_{xx}=6/2=3. (2) β^0=yˉ−β^1xˉ=113−3=23\hat\beta_0=\bar y-\hat\beta_1\bar x=\tfrac{11}{3}-3=\tfrac{2}{3}. (1) Fitted: y^={23, 113, 203}\hat y=\{\tfrac23,\ \tfrac{11}{3},\ \tfrac{20}{3}\}. Residuals: {1−23, 3−113, 7−203}={13,−23,13}\{1-\tfrac23,\ 3-\tfrac{11}{3},\ 7-\tfrac{20}{3}\}=\{\tfrac13,-\tfrac23,\tfrac13\}. (1) SSres=19+49+19=69=23SS_{res}=\tfrac19+\tfrac49+\tfrac19=\tfrac69=\tfrac23. SStot=∑(y−yˉ)2=649+49+1009=1689SS_{tot}=\sum(y-\bar y)^2=\tfrac{64}{9}+\tfrac{4}{9}+\tfrac{100}{9}=\tfrac{168}{9}. R2=1−2/3168/9=1−6168=1−128=2728≈0.9643.R^2=1-\dfrac{2/3}{168/9}=1-\dfrac{6}{168}=1-\dfrac{1}{28}=\dfrac{27}{28}\approx0.9643. (2)

Question 2

(a) [4] σ(z)=(1+e−z)−1\sigma(z)=(1+e^{-z})^{-1}. σ′(z)=e−z(1+e−z)2=σ(z)⋅e−z1+e−z=σ(z)(1−σ(z))\sigma'(z)=\dfrac{e^{-z}}{(1+e^{-z})^2}=\sigma(z)\cdot\dfrac{e^{-z}}{1+e^{-z}}=\sigma(z)(1-\sigma(z)). (2) σ(−z)=11+ez=e−ze−z+1=1−σ(z)\sigma(-z)=\dfrac{1}{1+e^{z}}=\dfrac{e^{-z}}{e^{-z}+1}=1-\sigma(z). (2)

(b) [8] L(w)=−∑i[yiln⁡p^i+(1−yi)ln⁡(1−p^i)]L(\mathbf w)=-\sum_{i}\big[y_i\ln\hat p_i+(1-y_i)\ln(1-\hat p_i)\big]. (2) Let zi=w⊤xiz_i=\mathbf w^\top\mathbf x_i, p^i=σ(zi)\hat p_i=\sigma(z_i). ∂L∂p^i=−yip^i+1−yi1−p^i=p^i−yip^i(1−p^i)\dfrac{\partial L}{\partial \hat p_i}=-\dfrac{y_i}{\hat p_i}+\dfrac{1-y_i}{1-\hat p_i}=\dfrac{\hat p_i-y_i}{\hat p_i(1-\hat p_i)}. (2) ∂p^i∂zi=p^i(1−p^i)\dfrac{\partial\hat p_i}{\partial z_i}=\hat p_i(1-\hat p_i), ∂zi∂w=xi\dfrac{\partial z_i}{\partial\mathbf w}=\mathbf x_i. Chain rule: ∇wL=∑ip^i−yip^i(1−p^i)⋅p^i(1−p^i) xi=∑i(p^i−yi)xi\nabla_{\mathbf w}L=\sum_i \dfrac{\hat p_i-y_i}{\hat p_i(1-\hat p_i)}\cdot\hat p_i(1-\hat p_i)\,\mathbf x_i=\sum_i(\hat p_i-y_i)\mathbf x_i. (3) Identical form to linear regression's ∑(y^i−yi)xi\sum(\hat y_i-y_i)\mathbf x_i — because the sigmoid is the canonical link of the Bernoulli GLM; the p^(1−p^)\hat p(1-\hat p) terms cancel exactly. (1)

(c) [4] (i) Boundary: p^=0.5⇒z=0\hat p=0.5\Rightarrow z=0: −3+2x1+x2=0⇒x2=3−2x1-3+2x_1+x_2=0\Rightarrow x_2=3-2x_1; slope =−2=-2. (2) (ii) z=−3+2(1)+1(2)=1z=-3+2(1)+1(2)=1; p^=σ(1)=11+e−1≈0.731\hat p=\sigma(1)=\dfrac{1}{1+e^{-1}}\approx0.731; >0.5⇒>0.5\Rightarrow class 1. (2)

(d) [4] Log-odds =ln⁡p1−p=w⊤x=\ln\dfrac{p}{1-p}=\mathbf w^\top\mathbf x. A unit increase in x1x_1 raises log-odds by w1=2w_1=2, so odds multiply by e2≈7.389e^{2}\approx7.389. (4) (The odds of the positive class become ~7.39× larger, x2x_2 fixed.)

Question 3

(a) [6] J(β)=12n∥y−Xβ∥22+λ(α∥β∥1+1−α2∥β∥22)J(\boldsymbol\beta)=\tfrac{1}{2n}\lVert\mathbf y-X\boldsymbol\beta\rVert_2^2+\lambda\big(\alpha\lVert\boldsymbol\beta\rVert_1+\tfrac{1-\alpha}{2}\lVert\boldsymbol\beta\rVert_2^2\big). (2) L1 constraint region {∣β1∣+∣β2∣≤t}\{|\beta_1|+|\beta_2|\le t\} is a diamond with sharp vertices lying on the axes; the elliptical loss contours typically first touch this region at a vertex, where one coordinate is exactly 0 → sparsity. (2) L2 region {β12+β22≤t}\{\beta_1^2+\beta_2^2\le t\} is a smooth circle with no corners; the tangent contact point generically has both coordinates nonzero, so Ridge shrinks but rarely zeroes. (2)

(b) [5] (i) Overfitting: degree-9 with 12 points has near-interpolation, high variance, low bias on training but large generalization error. (2) (ii) Any two, e.g.: (1) Reduce polynomial degree / model complexity → lowers variance. (2) Add L2/L1 regularization → penalizes large coefficients, shrinks variance. (3) Get more training data → reduces variance of fit. (4) Cross-validation to select degree/λ. (3)

(c) [4] Radj2=1−(1−R2)n−1n−p−1=1−(1−0.82)4941=1−0.18×1.19512=1−0.21512=0.7849.R^2_{adj}=1-(1-R^2)\dfrac{n-1}{n-p-1}=1-(1-0.82)\dfrac{49}{41}=1-0.18\times1.19512=1-0.21512=0.7849. (3) Adjusted R2R^2 penalizes extra predictors; unlike R2R^2 (which never decreases when predictors are added) it can fall, giving fair model comparison. (1)

(d) [3] Any three (1 each):

  • Linearity — violation → systematic curvature in residual plot.
  • Independence of errors — violation (autocorrelation) → underestimated SEs; Durbin–Watson test.
  • Homoscedasticity — violation → funnel-shaped residuals; inefficient estimates.
  • Normality of errors — affects inference/CIs; Q–Q plot.
  • No multicollinearity — high VIF → unstable coefficients.
[
  {"claim":"OLS fit of {(0,1),(1,3),(2,7)} gives beta1=3, beta0=2/3, R2=27/28",
   "code":"import numpy as np; X=Matrix([[1,0],[1,1],[1,2]]); y=Matrix([1,3,7]); b=(X.T*X).inv()*X.T*y; b0,b1=b[0],b[1]; yhat=X*b; res=y-yhat; ybar=Rational(11,3); ssres=sum([r**2 for r in res]); sstot=sum([(yi-ybar)**2 for yi in y]); R2=1-ssres/sstot; result=(b1==3) and (b0==Rational(2,3)) and (R2==Rational(27,28))"},
  {"claim":"sigma(1) approx 0.7310585",
   "code":"val=1/(1+exp(-1)); result=abs(float(val)-0.7310585786)<1e-6"},
  {"claim":"odds multiplier for w1=2 is e^2 approx 7.389056",
   "code":"result=abs(float(exp(2))-7.389056099)<1e-6"},
  {"claim":"adjusted R2 for R2=0.82,n=50,p=8 approx 0.78488",
   "code":"R2=Rational(82,100); n=50; p=8; adj=1-(1-R2)*Rational(n-1,n-p-1); result=abs(float(adj)-0.784878)<1e-5"}
]