Simple linear regression model

Overview

Simple linear regression models the relationship between a single independent variable $x$ and a dependent variable $y$ using a straight line. It's the foundation of predictive modeling and the gateway to understanding all supervised learning.

[!intuition] The Core Idea

Imagine you're predicting house prices. You notice: bigger houses cost more. Simple linear regression draws the "best-fit" straight line through your (size, price) data points. Once you have this line, you predict new prices by reading off the line.

Why a straight line? Because we're modeling a proportional relationship: each extra square foot adds roughly the same amount to the price. The world isn't always linear, but linear models are:

• Easy to interpret (slope = "price per sq ft")
• Fast to compute
• Surprisingly effective for many real phenomena

[!definition] Mathematical Model

The simple linear regression model is:

$y = \beta_0 + \beta_1 x + \epsilon$

Where:

• $y$ dependent variable (target, what we predict)
• $x$ = independent variable (feature, predictor)
• $\beta_0$ = intercept (value of $y$ when $x=0$)
• $\beta_1$ = slope (change in $y$ per unit change in $x$)
• $\epsilon$ = error term (random noise, captures what the line mises)

Key assumptions:

1. Linearity: True relationship is roughly linear
2. Independence: Observations don't affect each other
3. Homoscedasticity: Error variance is constant across all $x$
4. Normality: Errors $\epsilon$ follow $\mathcal{N}(0, \sigma^2)$

[!formula] Deriving the Best-Fit Line (Ordinary Least Squares)

Goal: Find $\beta_0, \beta_1$ that minimize prediction errors.

Step 1: Define the Loss Function

For $n$ training points $(x_i, y_i)$, our prediction is $\hat{y}_i = \beta_0 + \beta_1 x_i$.

The residual (error) for point $i$ is: $e_i = y_i - \hat{y}_i$

We want small errors, so minimize the sum of squared residuals (RSS):

$\text{RSS}(\beta_0, \beta_1) = \sum_{i=1}^{n} (y_i - \beta_0 - \beta_1 x_i)^2$

Why square the errors?

1. Squaring makes all errors positive (a $-5$ error is as bad as $+5$)
2. Squaring penalizes large errors more (outliers hurt)
3. Squaring makes the math differentiable (we can use calculus)

Step 2: Take Partial Derivatives

To minimize, set $\frac{\partial \text{RSS}}{\partial \beta_0} = 0$ and $\frac{\partial \text{RSS}}{\partial \beta_1} = 0$.

Derivative with respect to $\beta_0$: $\frac{\partial \text{RSS}}{\partial \beta_0} = \frac{\partial}{\partial \beta_0} \sum_{i=1}^{n} (y_i - \beta_0 - \beta_1 x_i)^2$

Using chain rule: $\frac{\partial}{\partial \beta_0}(y_i - \beta_0 - \beta_1 x_i)^2 = 2(y_i - \beta_0 - \beta_1 x_i) \cdot (-1)$

$= -2 \sum_{i=1}^{n} (y_i - \beta_0 - \beta_1 x_i) = 0$

Divide by $-2$: $\sum_{i=1}^{n} (y_i - \beta_0 - \beta_1 x_i) = 0$

Expand: $\sum_{i=1}^{n} y_i - n\beta_0 - \beta_1 \sum_{i=1}^{n} x_i = 0$

Why this step? Suming all residuals and setting to zero means the line goes through the "center" of the data.

Solve for $\beta_0$: $\beta_0 = \frac{1}{n}\sum_{i=1}^{n} y_i - \beta_1 \frac{1}{n}\sum_{i=1}^{n} x_i = \bar{y} - \beta_1 \bar{x}$

where $\bar{x} = \frac{1}{n}\sum x_i$ and $\bar{y} = \frac{1}{n}\sum y_i$ are the means.

Derivative with respect to $\beta_1$: $\frac{\partial \text{RSS}}{\partial \beta_1} = -2 \sum_{i=1}^{n} (y_i - \beta_0 - \beta_1 x_i) \cdot x_i = 0$

$\sum_{i=1}^{n} x_i(y_i - \beta_0 - \beta_1 x_i) = 0$

Substitute $\beta_0 = \bar{y} - \beta_1 \bar{x}$: $\sum_{i=1}^{n} x_i[y_i - (\bar{y} - \beta_1 \bar{x}) - \beta_1 x_i] = 0$

$\sum_{i=1}^{n} x_i(y_i - \bar{y}) - \beta_1 \sum_{i=1}^{n} x_i(x_i - \bar{x}) = 0$

Why this step? We're centering the data around its mean, which simplifies to covariance and variance.

Note: $\sum x_i \bar{x} = \bar \sum x_i = n\bar{x}^2$, and $\sum x_i y_i - \sum x_i \bar{y} = \sum x_i(y_i - \bar{y})$

Solve for $\beta_1$: $\beta_1 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n} (x_i - \bar{x})^2}$

In terms of statistics: $\beta_1 = \frac{\text{Cov}(x,y)}{\text{Var}(x)} = \frac{s_{xy}}{s_x^2}$

Final OLS formulas: $\boxed{\beta_1 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n} (x_i - \bar{x})^2}}$

$\boxed{\beta_0 = \bar{y} - \beta_1 \bar{x}}$

[!example] Worked Example 1: House Prices

Data: 5 houses with size (100s of sq ft) and price (1000s of $)

Step 1: Calculate means $\bar{x} = \frac{10+15+20+25+30}{5} = 20$ $\bar{y} = \frac{250+350+450+550+650}{5} = 450$

Step 2: Calculate slope $\beta_1$

$\beta_1 = \frac{5000}{250} = 20$

Why this value? Each100 sq ft increase raises price by $20k. Makes intuitive sense!

Step 3: Calculate intercept $\beta_0$ $\beta_0 = 450 - 20 \times 20 = 450 - 400 = 50$

Final model: $\hat{y} = 50 + 20x$

Interpretation: A house with 0 sq ft (doesn't exist!) would cost $50k (maybe land value?). Each 100 sq ft adds$20k.

Prediction: For a 200 sq ft house ($x=22$): $\hat{y} = 50 + 20(22) = 490 \text{ thousand dollars}$

[!example] Worked Example 2: Study Hours vs Exam Score

Data: 4 students

Step 1: $\bar{x} = 5$, $\bar{y} = 72.5$

Step 2:

$\sum(x_i-\bar{x})(y_i-\bar{y}) = (-3)(-22.5) + (-1)(-7.5) + (1)(7.5) + (3)(22.5)$ $= 67.5 + 7.5 + 67.5 = 150$

$\sum(x_i-\bar{x})^2 = 9 + 1 + 1 + 9 = 20$

$\beta_1 = \frac{150}{20} = 7.5$

Step 3: $\beta_0 = 72.5 - 7.5(5) = 35$

Model: $\hat{y} = 35 + 7.5x$

Meaning: Each extra study hour adds 7.5 points. With 0 hours, base score is 35 (prior knowledge?).

[!formula] Model Evaluation Metrics

1. R-squared (Coefficient of Determination)

$R^2 = 1 - \frac{\text{RSS}}{\text{TSS}} = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2}$

where TSS (Total Sum of Squares) = $\sum(y_i - \bar{y})^2$ measures total variance in $y$.

Derivation of meaning:

• RSS = unexplained variance (what the model missed)
• TSS = total variance in data
• $\frac{\text{RSS}}{\text{TS}}$ = fraction unexplained
• $R^2$ = fraction explained (between 0 and 1)

Why this metric? $R^2 = 0.85$ means "85% of variance in $y$ is explained by $x$". Higher is better.

2. Root Mean Squared Error (RMSE)

$\text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2}$

Why this metric? RMSE is in the same units as $y$. If predicting price in dollars, RMSE is in dollars. "On average, predictions are off by $±\text{RMSE}$."

[!mistake] Common Pitfalls

Mistake 1: Extrapolation Beyond Data Range

Wrong thinking: "My model works for houses1000-3000 sq ft, so I'll predict a10,000 sq ft mansion."

Why it feels right: The formula $\hat{y} = \beta_0 + \beta_1 x$ works for any $x$!

The fix: Linearity might break outside your data range. Huge houses have different price dynamics (luxury market). Only interpolate within observed $x$ range.

Mistake 2: Correlation≠ Causation

Wrong thinking: "Ice cream sales and drowning deaths are correlated and fit a linear model, so ice cream causes drowning!"

Why it feels right: The math is valid; the regression line exists.

The fix: Regression finds association, not causation. Both are caused by a confounding variable (summer heat). Need controlled experiments or causal inference methods to claim causation.

Mistake 3: Ignoring Outliers

Wrong thinking: "One data point is way off, but I'll include it anyway."

Why it feels right: More data is better, right?

The fix: Outliers distort the line (squared errors amplify their influence). Investigate outliers: data entry errors? Or legitimate extreme cases? Consider robust regression methods or remove after justification.

Mistake 4: Assuming Linearity Without Checking

Wrong thinking: "I fit a line, so the relationship is linear."

Why it feels right: You get a line no matter what; OLS always returns $\beta_0, \beta_1$.

The fix: Plot residuals vs fitted values. Random scatter = good. Patterns (curve, funel shape) = violated assumptions. Transform data (log, sqrt) or use polynomial/nonlinear regression.

[!recall]- Explain Like I'm 12

Imagine you're trying to guess how tall your friends are by looking at their shoe size.

You notice: bigger shoes usually mean taller person. So you draw a straight line on graph paper where you've ploted everyone's (shoe size, height).

Now when a new kid with size 7 shoes shows up, you find7 on your line and read off their predicted height!

Simple linear regression is just finding the BEST line through your dots. "Best" means the line that's closest to all dots at once (smallest mistakes on average).

The slope tells you "how many inches taller per shoe size", and the intercept is where the line crosses the height axis (though baby with size 0 shoes wouldn't actually be that height—the line is just a model!).

[!mnemonic] SLOPE

Sum of (x - mean)(y - mean)
Less
Over
Product of (x - mean)²
Equals beta₁

Intercept: "y-bar minus beta times x-bar" (sounds like "why bother? Beat times X!")

Connections

• 2.1-Introduction-to-Regression: Broader context
• 2.2.02-Multiple-Linear-Regression: Extension to multiple features
• 2.3-Gradient-Descent-for-Linear-Regression: Iterative optimization approach
• 2.5-Assumptions-and-Diagnostics: Checking model validity
• 3.1-Logistic-Regression-Model: Extension to classification
• 4.2-Overfitting-vs-Underfitting: Bias-variance in linear models
• 1.3-Loss-Functions: RSS is a specific loss function

#flashcards/ai-ml

What is simple linear regression modeling? :: A method to model the relationship between a single independent variable x and dependent variable y using a straight line equation y = β₀ + β₁x + ε

Derive the OLS formula for β₁ from first principles :: Take ∂RSS/∂β₁ = 0, substitute β₀ = ȳ - β₁x̄, solve to get β₁ = Σ(xᵢ-x̄)(yᵢ-ȳ) / Σ(xᵢ-x̄)² = Cov(x,y)/Var(x)

Concept Map

Hinglish (regional understanding)