Visual walkthrough — Linear regression — least squares, inference on coefficients

4.9.22 · D2 · Maths › Probability Theory & Statistics › Linear regression — least squares, inference on coefficients

Step 1 — Dots aur guess

KYA. Hum kuch measured pairs se shuru karte hain: har observation $i$ ke liye ek horizontal number $x_i$ aur ek vertical number $y_i$ hota hai. Inhe plot karne se ek point ka cloud banta hai. Hume lagta hai ki yeh cloud secretly ek straight line follow karta hai, lekin noise ne har dot ko thoda hila diya hai.

KYUN. Ek straight line sabse simple possible "kahaani" hai: ek starting height aur ek steepness chuno, aur tumne sirf do numbers se infinitely many points describe kar diye. Pehle hum koi bhi line draw aur score kar sakein, tabhi best line dhundh sakte hain.

PICTURE. Neeche, black dots data hain. Blue line ek candidate hai — ek guess. Iske do knobs hain:

$\hat{y}=\underset{\text{height where line crosses }x=0}{\underbrace{{\hat{\beta }}_0}}+\underset{\text{steepness: rise per 1 step right}}{\underbrace{{\hat{\beta }}_1}}\cdot x$

Step 2 — Miss measure karna: residual

KYA. Har dot ke liye, dot se line tak ek vertical segment giraao. Uski signed length hi residual hai:

$e_i=\underset{\text{where the dot IS}}{\underbrace{y_i}}-\underset{\text{where the line SAYS it should be}}{\underbrace{{\hat{y}}_i}}$

Vertical kyun, perpendicular kyun nahi? Hum $y$ ko $x$ se predict karne ki koshish kar rahe hain. Ek diya hua $x$ leke, line humein ek predicted $y$ deti hai. Honest error woh gap hai jo hum predict karte hain — $y$-direction mein. Perpendicular distance mein $x$ ki errors bhi aa jaati hain (jinhe hum known maante hain), jo score mein mix ho jaatein.

PICTURE. Orange segments misses hain. Line ke upar wale dots ka $e_i>0$ hai; neeche wale dots ka $e_i<0$ hai.

Step 3 — Misses ko square kyun karte hain

KYA. Hum ek line ko sum of squared residuals se score karte hain:

$S({\hat{\beta }}_0,{\hat{\beta }}_1)={\sum }_{i=1}^n{e}_i^2={\sum }_{i=1}^n(\underset{\text{the miss for dot }i}{\underbrace{y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i}}{)}^2$

Squaring kyun, absolute value kyun nahi? Teen reasons, har ek visible:

1. Squaring har miss ko positive bana deta hai, toh kuch cancel nahi hota.
2. Squaring badi misses ko zyada punish karta hai — $2$ ki miss $4$ cost karti hai, $4$ ki miss $16$ cost karti hai. Line apne worst offences se sabse zyada khinchti hai.
3. Squaring smooth hai: score curve mein koi sharp corner nahi, toh calculus (Step 5) bottom cleanly dhundh sakta hai. Absolute value mein kink hoti hai aur koi tidy formula nahi milta.

PICTURE. Length $e_i$ ka har orange segment ek orange square ban jaata hai jiska area $e_i^2$ hai. Total shaded area hi score $S$ hai. Least squares = total orange area ko shrink karo.

Step 4 — Score ek bowl mein rehta hai

KYA. $S$ do knobs ${\hat{\beta }}_0,{\hat{\beta }}_1$ pe depend karta hai. Un do numbers ko ek location maano aur $S$ ko unke upar height ki tarah plot karo. Kyunki $S$ squares ka sum hai, woh surface ek smooth upward bowl (paraboloid) hai jiska exactly ek lowest point hai.

KYUN important hai. Ek bowl ka ek hi bottom hota hai, aur us bottom pe zameen har direction mein flat hoti hai. "Flat" ek aisi cheez hai jo calculus exactly pin down kar sakta hai. Yahi guarantee karta hai ek unique best line — koi doosra candidate nahi, koi ambiguity nahi.

PICTURE. $({\hat{\beta }}_0,{\hat{\beta }}_1)$ plane ke upar bowl. Red dot minimum hai; wahan red tangent plane horizontal hai.

Step 5 — Bowl ki do slopes zero set karo

KYA. Hum $S$ ka derivative har knob ke respect mein lete hain aur dono ko zero set karte hain. Derivative $\partial S/\partial {\hat{\beta }}_0$ poochta hai "agar main height ko thoda nudge karun, toh kya score badlega?"; bottom pe answer hona chahiye "nahi."