Interpreting model coefficients

WHY do we care about coefficients at all?

We don't just want a black box that predicts. We want to understand: which features matter, in which direction, and by how much. A coefficient $\beta_j$ is the model's estimated slope along feature $x_j$. Interpreting it correctly is the difference between "smoking increases risk" and a meaningless number.

WHAT does a LINEAR regression coefficient mean?

Model: $\hat{y} = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p$.

Derivation of interpretation (from scratch). Take the prediction at $x_j$ and at $x_j + 1$ (everything else fixed):

$\hat{y}(x_j+1) - \hat{y}(x_j) = \big[\beta_0 + \dots + \beta_j(x_j+1) + \dots\big] - \big[\beta_0 + \dots + \beta_j x_j + \dots\big] = \beta_j.$

• Intercept $\beta_0$: predicted $y$ when all features are 0 (often meaningless if $x=0$ is out of range — that's why we sometimes center features).

WHAT does a LOGISTIC regression coefficient mean?

Logistic regression models log-odds, not probability:

$\log\!\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p, \qquad p = P(y=1\mid x).$

Derivation of the odds-ratio interpretation. The log-odds is linear, so as before:

$\text{log-odds}(x_j+1) - \text{log-odds}(x_j) = \beta_j.$

Exponentiate both sides. Let $\text{odds}=\frac{p}{1-p}$:

$\frac{\text{odds}(x_j+1)}{\text{odds}(x_j)} = e^{\beta_j}.$

HOW to compare coefficients: standardization

You cannot say "$\beta_1 > \beta_2$ so feature 1 is more important" if the features are on different scales (income in $ vs. age in years). A tiny coefficient on income might dominate because income values are huge.

Why $\beta_j^{\text{std}} = \beta_j\sigma_j$? If $z_j = x_j/\sigma_j$ (roughly), then a 1-unit change in $z_j$ is a $\sigma_j$-unit change in $x_j$, giving $\beta_j\sigma_j$ change in $y$.

Common mistakes (Steel-manned)

Active recall

Connections

• Linear Regression — where additive coefficients come from
• Logistic Regression — log-odds and the sigmoid
• Odds and Log-Odds — the transform behind $e^\beta$
• Feature Scaling & Standardization — enabling fair comparison
• Multicollinearity & VIF — why "holding others fixed" can break down
• Regularization (Ridge & Lasso) — stabilizing/shrinking coefficients
• Correlation vs Causation

Concept Map

Hinglish (regional understanding)