Linear regression — least squares, inference on coefficients: Exercises

4.9.22 · D4 · Maths › Probability Theory & Statistics › Linear regression — least squares, inference on coefficients

Shuru karne se pehle, hum apna toolbox define kar lete hain (koi bhi cheez use karne se pehle naam de di jaayegi):

Kuch problems mein ek shared dataset (isko Set A bolenge) use hota hai taaki ek hi table par fluency bane:

$x = (1, 2, 3, 4, 5, 6), y = (3, 4, 6, 5, 8, 9) .$

Figure mein Set A apni least-squares line (cyan) ke saath dikhaya gaya hai, centroid $(\overset{x}{ˉ}, \overset{y}{ˉ})$ (amber cross) ke saath, aur residuals $e_{i}$ amber vertical segments ke roop mein. L2 aur L3 karte waqt ise khula rakho — line woh hai jo tum fit karte ho, segments woh hain jo tum SSE pane ke liye square karte ho.

Figure — Linear regression — least squares, inference on coefficients

Level 1 — Recognition

L1.1 — Ek line se pieces padho

Kuch data ka fitted line hai $\overset{y}{^} = 2.2 + 0.6 x$ . (a) Estimated slope $\hat{β}_{1}$ kya hai? (b) Estimated intercept $\hat{β}_{0}$ kya hai? (c) Slope ka matlab words mein kya hota hai?

Recall Solution

(a) Slope woh number hai jo $x$ ko multiply kar raha hai: $\hat{β}_{1} = 0.6$ . (b) Intercept woh standalone number hai (jab $x = 0$ ho tab ki value): $\hat{β}_{0} = 2.2$ . (c) "Jab bhi $x$ $1$ se badhta hai, fitted $y$ average par $0.6$ se badhta hai." Slope ek rate hai, total nahi.

L1.2 — Kaun sa number kaun sa hai?

Set A ke liye bataya gaya hai ki $\overset{x}{ˉ} = 3.5$ . $\overset{y}{ˉ}$ compute karo.

Recall Solution

$\overset{y}{ˉ} = \frac{3 + 4 + 6 + 5 + 8 + 9}{6} = \frac{35}{6} \approx 5.833$ . Mean sirf add-karke-divide hai; yahan $n = 6$ hai.

Level 2 — Application

L2.1 — Set A par haath se line fit karo

$x = (1, 2, 3, 4, 5, 6)$ , $y = (3, 4, 6, 5, 8, 9)$ use karke $\overset{x}{ˉ}, \overset{y}{ˉ}, S_{xx}, S_{x y}$ compute karo, phir $\hat{β}_{1}$ aur $\hat{β}_{0}$ .

Recall Solution

Pehle Means (sab kuch inhi par centre hota hai): $\overset{x}{ˉ} = 21/6 = 3.5$ , $\overset{y}{ˉ} = 35/6 \approx 5.8333$ .

Centred $x$ deviations $(x_{i} - \overset{x}{ˉ})$ : $- 2.5, - 1.5, - 0.5, 0.5, 1.5, 2.5$ . $S_{xx} = 2. 5^{2} + 1. 5^{2} + 0. 5^{2} + 0. 5^{2} + 1. 5^{2} + 2. 5^{2} = 6.25 + 2.25 + 0.25 + 0.25 + 2.25 + 6.25 = 17.5.$

Centred $y$ deviations $(y_{i} - \overset{y}{ˉ})$ : $- 2.8333, - 1.8333, 0.1667, - 0.8333, 2.1667, 3.1667$ . Pairwise multiply karke add karo: $S_{x y} = (- 2.5) (- 2.8333) + (- 1.5) (- 1.8333) + (- 0.5) (0.1667)$ $+ (0.5) (- 0.8333) + (1.5) (2.1667) + (2.5) (3.1667) = 20.5.$ (Term by term: $7.0833 + 2.75 - 0.0833 - 0.4167 + 3.25 + 7.9167 = 20.5$ .)

Slope = co-movement divided by $x$ -spread: $\hat{β}_{1} = \frac{20.5}{17.5} \approx 1.1714$ . Intercept (line ko $(\overset{x}{ˉ}, \overset{y}{ˉ})$ se guzarata hai): $\hat{β}_{0} = 5.8333 - 1.1714 (3.5) \approx 1.7333$ . Fitted line: $\overset{y}{^} \approx 1.733 + 1.171 x$ .

L2.2 — Centroid par predict karo aur check karo

L2.1 ki line use karke $x = 4$ par $\overset{y}{^}$ predict karo, aur verify karo ki line $(\overset{x}{ˉ}, \overset{y}{ˉ}) = (3.5, 5.8333)$ se guzarti hai.

Recall Solution

$x = 4$ par: $\overset{y}{^} = 1.7333 + 1.1714 (4) = 1.7333 + 4.6857 = 6.4190$ . $x = 3.5$ par: $\overset{y}{^} = 1.7333 + 1.1714 (3.5) = 1.7333 + 4.1 = 5.8333 = \overset{y}{ˉ}$ ✓. Least-squares line hamesha centre of mass se guzarti hai — ek free sanity check.

Level 3 — Analysis

L3.1 — Standard error aur $t$ -statistic

Set A ke liye ( $n = 6$ ) residuals $e_{i} = y_{i} - \overset{y}{^}_{i}$ hain aur residual sum of squares $SSE = \sum e_{i}^{2} = 7.31429$ hai (yeh given maan lo). $s^{2}$ , $s$ , $SE (\hat{β}_{1})$ , aur $β_{1} = 0$ test karne ka $t$ -statistic compute karo. (Ye inference steps upar toolbox mein bataye gaye error conditions assume karte hain.)

Recall Solution

Residual variance (do coefficients estimate hue isliye $n - 2$ se divide karo): $s^{2} = \frac{SSE}{n - 2} = \frac{7.31429}{4} = 1.82857, s = 1.82857 \approx 1.35225.$ Standard error of the slope (toolbox mein define hai — zyada $x$ -spread se precision badhti hai): $SE (\hat{β}_{1}) = \frac{s}{S _{xx}} = \frac{1.35225}{17.5} = \frac{1.35225}{4.18330} \approx 0.32324.$ $t$ -statistic (slope zero se kitne SEs dur hai): $t = \frac{β ^ _{1} - 0}{SE ( β ^ _{1} )} = \frac{1.17143}{0.32324} \approx 3.624 on n - 2 = 4 df .$

L3.2 — Significance decide karo

Critical value $t_{4, 0.975} = 2.776$ . 5% level par, kya hum $β_{1} = 0$ reject karte hain? 95% CI banao.

Recall Solution

$∣ t ∣ = 3.624 > 2.776$ , isliye hum $β_{1} = 0$ reject karte hain: slope significant hai. 95% CI: $\hat{β}_{1} \pm t_{4, 0.975} \cdot SE = 1.17143 \pm 2.776 (0.32324)$ $= 1.17143 \pm 0.89732 = [0.274, 2.069]$ . Interval mein $0$ nahi hai — rejection ke saath consistent hai. Notice karo ki yeh same decision do angles se aa raha hai (Hypothesis Testing $t$ se, ya t-distribution ke saath $4$ df wale CI se).

Level 4 — Synthesis

L4.1 — Decomposition se $R^{2}$

Set A ke liye, $S_{y y} = SST = 31.3333$ aur $SSE = 7.31429$ hai. $R^{2}$ do tarike se compute karo (SSE ke zariye aur SSR ke zariye) aur confirm karo ki $R^{2} = r^{2}$ , jahan $r$ Correlation coefficient hai.

Recall Solution

Total up–down variation: $SST = 31.3333$ . Residual (unexplained): $SSE = 7.31429$ . Explained: $SSR = SST - SSE = 31.3333 - 7.31429 = 24.0190$ .

R^2=\frac{\text{SSR}}{\text{SST}}=\frac{24.0190}{31.3333}=0.7666.\ \checkmark$$ **Correlation check (exact):** $r^2 = \dfrac{S_{xy}^2}{S_{xx}S_{yy}}=\dfrac{20.5^2}{17.5\cdot31.3333}=\dfrac{420.25}{548.33}=0.7664.$ $0.7666$ se thoda sa gap sirf "given" SSE mein rounding ki wajah se hai; exact arithmetic se $R^2=r^2$ identically milta hai. Line $y$ ki variance ka lagbhag **77\%** explain karta hai.

L4.2 — Lever effect forecast karo phir verify karo

Data se recompute kiye bina: suppose karo ki tum $x$ 's ka spread triple kar sako (toh $S_{xx}$ $9 \times$ bada ho jaaye) jabki $s$ same rahe. Predict karo ki $SE (\hat{β}_{1})$ ka kya hoga, phir formula se confirm karo. Physically iska kya matlab hai?

Recall Solution

Predict: $SE (\hat{β}_{1}) = s / S_{xx}$ . Agar $S_{xx} \to 9 S_{xx}$ , toh $S_{xx} \to 3 S_{xx}$ , isliye SE ko ek-tihaai ho jaana chahiye. Numerically verify karo: old SE $= 1.3523/ 17.5 = 0.3233$ ; new SE $= 1.3523/ 157.5 = 0.1078$ , aur $0.3233/0.1078 = 3.00$ ✓. Physical meaning: wide-spread $x$ values line ko lambi lever arm dete hain, isliye same vertical wobble slope ko kaafi zyada precisely pin karta hai — yeh Gauss–Markov Theorem ki us baat ki echo hai ki spread-out design points least-squares estimator ko sharp banate hain.

Level 5 — Mastery

L5.1 — Slope ki variance scratch se rebuild karo

$\hat{β}_{1} = \sum_{i} c_{i} y_{i}$ se shuru karo jahan weights $c_{i} = (x_{i} - \overset{x}{ˉ}) / S_{xx}$ hain, aur $Var (y_{i}) = σ^{2}$ independent errors ke saath use karke prove karo ki $Var (\hat{β}_{1}) = σ^{2} / S_{xx}$ . Dikhao ki $c_{i}$ itne neat kyun collapse hote hain.

Recall Solution

Hum kya karte hain: $\hat{β}_{1}$ ko random $y_{i}$ ka ek fixed weighted sum treat karo. Independent terms ke liye, sum ki variance, (weight $^{2} \times$ variance) ka sum hai: $Var (\hat{β}_{1}) = \sum_{i} c_{i}^{2} Var (y_{i}) = σ^{2} \sum_{i} c_{i}^{2} .$ Kyun: independence saare cross-covariance terms ko khatam kar deti hai — sirf diagonal bachti hai. Ab collapse:

=\frac{S_{xx}}{S_{xx}^2}=\frac{1}{S_{xx}}.$$ Isliye $\text{Var}(\hat\beta_1)=\dfrac{\sigma^2}{S_{xx}}$. Numerator sum-of-squares *hi* $S_{xx}$ hai, isliye ek power cancel ho jaati hai. Yeh "long lever" fact rigorous roop mein prove hua hai, aur $\sigma^2$ ki jagah $s^2$ daalne se $\text{SE}(\hat\beta_1)=s/\sqrt{S_{xx}}$ milta hai jo humne L3 mein assume kiya tha. (Gaussian errors ke under yeh variance, ek $t$-ratio mein plug hoke, wahi deta hai jo [[Maximum Likelihood Estimation]] bhi deta.)

L5.2 — Naye data par poora pipeline

Naya dataset Set B: $x = (0, 1, 2, 3, 4)$ , $y = (1, 1, 2, 2, 4)$ . Poora kaam karo: $\hat{β}_{1}, \hat{β}_{0}$ , $SSE$ , $s^{2}$ , $SE (\hat{β}_{1})$ , $β_{1} = 0$ ka $t$ -statistic, aur 5% par significance decision ( $t_{3, 0.975} = 3.182$ ).

Recall Solution

Means: $\overset{x}{ˉ} = 2$ , $\overset{y}{ˉ} = (1 + 1 + 2 + 2 + 4) /5 = 2$ . $x$ -deviations: $- 2, - 1, 0, 1, 2$ → $S_{xx} = 4 + 1 + 0 + 1 + 4 = 10$ . $y$ -deviations: $- 1, - 1, 0, 0, 2$ . $S_{x y} = (- 2) (- 1) + (- 1) (- 1) + 0 + 0 + (2) (2) = 2 + 1 + 0 + 0 + 4 = 7$ . Slope: $\hat{β}_{1} = 7/10 = 0.7$ . Intercept: $\hat{β}_{0} = 2 - 0.7 (2) = 0.6$ . Line $\overset{y}{^} = 0.6 + 0.7 x$ .

Fitted values: $0.6, 1.3, 2.0, 2.7, 3.4$ . Residuals $e_{i} = y_{i} - \overset{y}{^}_{i}$ : $0.4, - 0.3, 0.0, - 0.7, 0.6$ . $SSE = 0.16 + 0.09 + 0 + 0.49 + 0.36 = 1.10.$ $s^{2} = \frac{1.10}{5 - 2} = 0.3667$ , $s = 0.6055$ . $SE (\hat{β}_{1}) = \frac{0.6055}{10} = 0.1915$ . $t = \frac{0.7}{0.1915} = 3.656$ on $3$ df. Kyunki $3.656 > 3.182$ , hum 5% par $β_{1} = 0$ reject karte hain: sirf 5 points ke saath bhi slope significant hai, kyunki fit tight hai aur $x$ 's spread hain.

Recall Ek-line self-test

Least squares ka slope ::: $\hat{β}_{1} = S_{x y} / S_{xx}$ SSE ko $n - 2$ se kyun divide karte hain ::: do constraints ( $\sum e_{i} = 0$ , $\sum x_{i} e_{i} = 0$ ) 2 df hata dete hain, jisse $s^{2}$ unbiased ho jaata hai $SE (\hat{β}_{1})$ chhota kya banata hai ::: bada $S_{xx}$ (spread-out $x$ ) aur chhota $s$ (kam noise) $R^{2}$ ek phrase mein ::: $y$ ki variance ka woh fraction jo line explain karti hai, yahan $r^{2}$ ke barabar hai