Noesis
Level 2 — RecallLinear & Logistic Regression

Linear & Logistic Regression

⏱ 30 minutes40 marksprintable — key stays hidden on paper

Time Limit: 30 minutes
Total Marks: 40
Instructions: Answer all questions. Use ...... notation for math where required. Show working for derivations and numeric problems.

Q1. State the simple linear regression model equation and briefly define each term. (3 marks)

Q2. Write the Mean Squared Error (MSE) cost function J(θ)J(\theta) for linear regression over mm examples. State the gradient descent update rule for parameter θj\theta_j with learning rate α\alpha. (4 marks)

Q3. Given the data points (1,2),(2,2),(3,4)(1,2), (2,2), (3,4), use the ordinary least squares formulas to find the slope β1^\hat{\beta_1} and intercept β0^\hat{\beta_0} of the best-fit line. (6 marks)

Q4. State the normal equation closed-form solution for the parameter vector θ\theta in linear regression. Name one situation where it cannot be applied directly. (3 marks)

Q5. List four key assumptions of linear regression. (4 marks)

Q6. Define R2R^2 (coefficient of determination) in terms of SSresSS_{res} and SStotSS_{tot}. Write the formula for adjusted R2R^2 and explain why it is preferred when adding predictors. (5 marks)

Q7. Write the sigmoid function σ(z)\sigma(z). Compute σ(0)\sigma(0) and evaluate the class label predicted (using threshold 0.5) when z=−2z = -2. (4 marks)

Q8. Write the log-loss (binary cross-entropy) for a single training example with true label yy and predicted probability y^\hat{y}. Compute the loss when y=1y=1 and y^=0.8\hat{y}=0.8 (use ln⁡\ln). (4 marks)

Q9. Contrast L1 (Lasso) and L2 (Ridge) regularization: give the penalty term added to the cost for each, and state one distinguishing effect of each on coefficients. (4 marks)

Q10. In logistic regression, a coefficient βj=0.7\beta_j = 0.7. Interpret its meaning in terms of the odds. (3 marks)

End of Paper

Answer keyMark scheme & solutions

Q1. (3 marks)
Model: y=β0+β1x+ϵy = \beta_0 + \beta_1 x + \epsilon — 1 mark.

  • yy: dependent/response variable; xx: independent/predictor variable — 0.5
  • β0\beta_0: intercept, β1\beta_1: slope — 1
  • ϵ\epsilon: random error term (mean 0) — 0.5

Q2. (4 marks)
Cost: J(θ)=12m∑i=1m(hθ(x(i))−y(i))2J(\theta)=\dfrac{1}{2m}\sum_{i=1}^{m}\left(h_\theta(x^{(i)})-y^{(i)}\right)^2 — 2 marks (accept without the 12\tfrac12 factor if consistent).
Update: θj:=θj−α1m∑i=1m(hθ(x(i))−y(i))xj(i)\theta_j := \theta_j - \alpha\dfrac{1}{m}\sum_{i=1}^{m}\left(h_\theta(x^{(i)})-y^{(i)}\right)x_j^{(i)} — 2 marks.
Why: gradient descent moves parameters opposite the gradient of JJ to minimise error.

Q3. (6 marks)
Means: xˉ=2\bar x = 2, yˉ=83\bar y = \tfrac{8}{3} — 1 mark.
β^1=∑(xi−xˉ)(yi−yˉ)∑(xi−xˉ)2\hat\beta_1 = \dfrac{\sum (x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2} — 1.
Numerator: (−1)(2−83)+(0)(… )+(1)(4−83)=(−1)(−23)+0+(1)(43)=23+43=2(-1)(2-\tfrac83)+(0)(\dots)+(1)(4-\tfrac83)=(-1)(-\tfrac23)+0+(1)(\tfrac43)=\tfrac23+\tfrac43=2 — 1.5.
Denominator: (−1)2+02+12=2(-1)^2+0^2+1^2 = 2 — 1.
β^1=2/2=1\hat\beta_1 = 2/2 = 1 — 0.5.
β^0=yˉ−β^1xˉ=83−1⋅2=23≈0.667\hat\beta_0 = \bar y - \hat\beta_1\bar x = \tfrac83 - 1\cdot 2 = \tfrac23 \approx 0.667 — 1.
Line: y^=0.667+1⋅x\hat y = 0.667 + 1\cdot x.

Q4. (3 marks)
Normal equation: θ=(X⊤X)−1X⊤y\theta = (X^\top X)^{-1}X^\top y — 2 marks.
Cannot be applied when X⊤XX^\top X is non-invertible (singular) — e.g. redundant/linearly dependent features or when features >> samples — 1 mark.

Q5. (4 marks — 1 each)
Any four of:

  • Linearity (relationship between predictors and response is linear).
  • Independence of errors.
  • Homoscedasticity (constant error variance).
  • Normality of residuals.
  • No (perfect) multicollinearity among predictors.

Q6. (5 marks)
R2=1−SSresSStotR^2 = 1 - \dfrac{SS_{res}}{SS_{tot}} — 2 marks.
Adj R2=1−(1−R2)(n−1)n−p−1\text{Adj }R^2 = 1 - \dfrac{(1-R^2)(n-1)}{n-p-1}, where nn=samples, pp=predictors — 2 marks.
Preferred because it penalises adding predictors that don't improve the model; plain R2R^2 never decreases when adding variables — 1 mark.

Q7. (4 marks)
σ(z)=11+e−z\sigma(z)=\dfrac{1}{1+e^{-z}} — 1.5 marks.
σ(0)=11+1=0.5\sigma(0)=\dfrac{1}{1+1}=0.5 — 1.
σ(−2)=11+e2≈0.119<0.5\sigma(-2)=\dfrac{1}{1+e^{2}}\approx 0.119 < 0.5 → predicted class 00 — 1.5.

Q8. (4 marks)
L=−[yln⁡y^+(1−y)ln⁡(1−y^)]L=-\big[y\ln\hat y + (1-y)\ln(1-\hat y)\big] — 2 marks.
With y=1,y^=0.8y=1,\hat y=0.8: L=−ln⁡(0.8)≈0.223L=-\ln(0.8)\approx 0.223 — 2 marks.

Q9. (4 marks)
L1 (Lasso): penalty λ∑j∣θj∣\lambda\sum_j|\theta_j| — 1. Effect: drives some coefficients exactly to 0 → feature selection/sparsity — 1.
L2 (Ridge): penalty λ∑jθj2\lambda\sum_j\theta_j^2 — 1. Effect: shrinks coefficients toward 0 smoothly but rarely exactly to 0 — 1.

Q10. (3 marks)
In logistic regression the log-odds are linear in features. A coefficient βj=0.7\beta_j=0.7 means a one-unit increase in xjx_j multiplies the odds by e0.7≈2.01e^{0.7}\approx 2.01 — 2 marks, i.e. the odds roughly double, holding other features constant — 1 mark.

[
  {"claim":"OLS slope for (1,2),(2,2),(3,4) is 1","code":"xs=[1,2,3]; ys=[2,2,4]; xb=sum(xs)/3; yb=Rational(8,3); num=sum((x-xb)*(y-yb) for x,y in zip(xs,ys)); den=sum((x-xb)**2 for x in xs); b1=num/den; result = (b1==1)"},
  {"claim":"OLS intercept is 2/3","code":"xs=[1,2,3]; ys=[2,2,4]; xb=Integer(2); yb=Rational(8,3); b1=Integer(1); b0=yb-b1*xb; result = (b0==Rational(2,3))"},
  {"claim":"sigmoid(-2) is below 0.5 (predict class 0)","code":"z=-2; s=1/(1+exp(-z)); result = bool(s<Rational(1,2))"},
  {"claim":"log-loss for y=1,yhat=0.8 approx 0.2231","code":"L=-ln(Rational(8,10)); result = bool(abs(L-0.2231435513)<1e-6)"},
  {"claim":"exp(0.7) approx 2.0138 (odds roughly double)","code":"result = bool(abs(exp(Rational(7,10))-2.013752707)<1e-6)"}
]