Worked examples — Multiple linear regression

This page is the practice ground for the parent topic. There we built the model $\hat{\mathbf{y}}=\mathbf{X}\mathbf{\beta }$ and the Normal Equation $\mathbf{\beta }=({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$. Here we drive it into every corner: good data, degenerate data, singular matrices, sign traps, and word problems.

Before symbols, plain words for the actors:

• Feature $x_j$ — one measured number about an example (size of a house, hours studied).
• Target $y$ — the number we want to predict (price, exam score).
• $m$ = number of examples (rows of data, i.e. how many houses/students you measured).
• $n$ = number of features (how many measured quantities per example, excluding the intercept). So the number of unknown coefficients is $n+1$: one intercept ${\beta }_0$ plus one weight per feature.
• $\mathbf{X}$ — a table: one row per example ($m$ rows), one column per feature, plus a leading column of $1$s that carries the intercept ${\beta }_0$ (so $n+1$ columns total).
• ${\mathbf{X}}^{\top }$ ("X transpose") — the same table flipped so rows become columns. We need it because ${\mathbf{X}}^{\top }\mathbf{X}$ collects, for every pair of features, the sum of their products across all examples — a compact record of how features move together.
• Residual $e^{(i)}=y^{(i)}-{\hat{y}}^{(i)}$ — the miss on example $i$: true minus predicted.

Throughout, we use "Cell" (capitalized) to name each row of the scenario matrix below, and each example is tagged with the Cell it hits.

---

The scenario matrix

Every problem this topic can throw is one of these Cells. Here $m$ = number of examples (rows) and $n+1$ = number of coefficients (intercept plus one per feature).

Cell	What makes it special	Covered by
A. Exactly determined	$m=n+1$ (as many rows as coefficients); plane passes through all points, zero residual	Ex 1
B. Overdetermined (real fit)	$m>n+1$; a genuine best fit with nonzero residuals	Ex 2
C. Mixed signs	Some coefficients positive, some negative; interpreting direction	Ex 3
D. Zero / degenerate input	A feature that never varies, or predicting at $x=0$	Ex 4
E. Singular ${\mathbf{X}}^{\top }\mathbf{X}$ (multicollinearity)	One feature = linear combo of others; no unique $\mathbf{\beta }$	Ex 5
F. Underdetermined ($m<n+1$)	Fewer examples than coefficients; infinite solutions	Ex 6
G. Limiting behaviour	What happens as one feature grows without bound / dominant coefficient	Ex 7
H. Word problem	Translate a real story into $\mathbf{X},\mathbf{y}$ and predict	Ex 8
I. Exam twist	Regularization rescue when the clean formula dies	Ex 9
J. Nearly-singular (ill-conditioned)	${\mathbf{X}}^{\top }\mathbf{X}$ almost — but not exactly — non-invertible; numerically unstable	Ex 10

---

Example 1 — Cell A: exactly determined

Step 1 — build $\mathbf{X}$ with the $1$s column. $\mathbf{X}=\left[\begin{array}{ccc}1& 1500& 3\\ 1& 2000& 4\\ 1& 1800& 3\end{array}\right],\mathbf{y}=\left[\begin{array}{c}300\\ 400\\ 350\end{array}\right]$ Why this step? The leading $1$s let the intercept ride along as "coefficient of the constant feature," so one matrix product handles all three predictions.

Step 2 — solve the actual linear system, don't guess. With $m=n+1=3$ the square system $\mathbf{X}\mathbf{\beta }=\mathbf{y}$ has (at most) one solution. Write the three equations out:

\beta_0 + 1500\beta_1 + 3\beta_2 &= 300 \\ \beta_0 + 2000\beta_1 + 4\beta_2 &= 400 \\ \beta_0 + 1800\beta_1 + 3\beta_2 &= 350 \end{aligned}$$ *Why this step?* When $\mathbf{X}$ is square and invertible the Normal Equation collapses to just solving $\mathbf{X}\boldsymbol\beta=\mathbf{y}$ — the plane can hit every point exactly, so residuals are zero. **Step 3 — eliminate to find the coefficients.** Subtract row 1 from row 3: only the $\beta_1$ term survives because $\beta_2$'s coefficient ($3$) is identical: $$(1800-1500)\beta_1 = 350-300 \;\Rightarrow\; 300\,\beta_1 = 50 \;\Rightarrow\; \beta_1 = \tfrac{1}{6}.$$ *Why this step?* Rows 1 and 3 share bedroom count $3$, so differencing them isolates the pure size-effect $\beta_1$ — a clean geometric slice. **Step 4 — back-substitute.** Subtract row 1 from row 2: $500\beta_1 + \beta_2 = 100$. With $\beta_1=\tfrac16$: $\beta_2 = 100 - \tfrac{500}{6} = \tfrac{100}{6} = \tfrac{50}{3}$. Then from row 1: $\beta_0 = 300 - 1500(\tfrac16) - 3(\tfrac{50}{3}) = 300 - 250 - 50 = 0$. $$\boldsymbol\beta = \begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \end{bmatrix} = \begin{bmatrix} 0 \\ \tfrac16 \\ \tfrac{50}{3} \end{bmatrix} \approx \begin{bmatrix} 0 \\ 0.1667 \\ 16.67 \end{bmatrix}.$$ *Why this step?* Back-substitution walks the eliminated equations upward, pinning down one unknown at a time. **Step 5 — read the model.** $$\text{Price} = 0 + \tfrac16 x_1 + \tfrac{50}{3} x_2.$$ *Why this step?* Assembling the coefficients into a single equation turns three abstract numbers into a usable rule you can feed any new house into — the whole point of fitting. **Step 6 — predict (1600, 3):** $$\hat y = \tfrac16(1600) + \tfrac{50}{3}(3) = 266.\overline{6} + 50 = 316.\overline{6}\text{ k} \approx 316.7\text{ k}.$$ *Why this step?* Plugging a *new* input the model never saw is the payoff — we test whether the learned rule generalizes beyond the training rows. > **Verify:** Plug every row back. House 1: $\tfrac16(1500)+\tfrac{50}{3}(3)=250+50=300$ ✓. House 2: $\tfrac16(2000)+\tfrac{50}{3}(4)=333.\overline3+66.\overline6=400$ ✓. House 3: $\tfrac16(1800)+\tfrac{50}{3}(3)=300+50=350$ ✓. All residuals zero — exactly determined confirmed. SymPy check in VERIFY. > [!mistake] Don't trust a "clean-looking" answer without plugging back > A tempting tidy guess like $(-150, 0.2, 50)$ fits house 1 but misses houses 2 and 3. The lesson of Cell A: **always** substitute *every* row. Only the actual elimination above — giving $\boldsymbol\beta=(0,\tfrac16,\tfrac{50}{3})$ — hits all three. --- ## Example 2 — Cell B: overdetermined, real least-squares fit > [!example] Four points, two unknowns (one feature) > Fit $\hat y = \beta_0 + \beta_1 x$ to: > > | $x$ | 1 | 2 | 3 | 4 | > |---|---|---|---|---| > | $y$ | 2 | 3 | 5 | 4 | > > **Forecast:** Here $m=4$ examples but only $n+1=2$ coefficients, so $m>n+1$. Will the line pass through all points? **Step 1 — why overdetermined?** With $m=4 > 2$ coefficients, the line generally *cannot* touch all four points. We accept nonzero residuals and minimize their squared sum. This connects to [[Simple Linear Regression]]. **Step 2 — the Normal Equation for one feature, shown fully.** With $\mathbf{X}=\begin{bmatrix}1&x^{(1)}\\ \vdots&\vdots\end{bmatrix}$, the $2\times2$ matrix and right-hand side are $$\mathbf{X}^\top\mathbf{X} = \begin{bmatrix} m & \sum x \\ \sum x & \sum x^2 \end{bmatrix}, \quad \mathbf{X}^\top\mathbf{y} = \begin{bmatrix} \sum y \\ \sum xy \end{bmatrix}.$$ *Why this step?* This is the general $(\mathbf{X}^\top\mathbf{X})\boldsymbol\beta=\mathbf{X}^\top\mathbf{y}$ written out for one feature, so you *see* what the abstract formula becomes. **Step 3 — fill in the sums.** $\sum x = 10,\ \sum x^2 = 30,\ \sum y = 14,\ \sum xy = 1(2)+2(3)+3(5)+4(4)=2+6+15+16=39$. $$\begin{bmatrix} 4 & 10 \\ 10 & 30 \end{bmatrix}\begin{bmatrix}\beta_0\\ \beta_1\end{bmatrix} = \begin{bmatrix} 14 \\ 39 \end{bmatrix}.$$ *Why this step?* Computing the four running sums turns the raw data into the exact numbers the Normal Equation needs — this is where the actual dataset enters the abstract formula. **Step 4 — invert the $2\times2$ explicitly.** For $\begin{bmatrix}a&b\\c&d\end{bmatrix}$ the inverse is $\tfrac{1}{ad-bc}\begin{bmatrix}d&-b\\-c&a\end{bmatrix}$. Determinant $= 4\cdot30 - 10\cdot10 = 120-100 = 20$. $$\boldsymbol\beta = \tfrac{1}{20}\begin{bmatrix}30 & -10 \\ -10 & 4\end{bmatrix}\begin{bmatrix}14\\39\end{bmatrix} = \tfrac{1}{20}\begin{bmatrix}30(14)-10(39)\\ -10(14)+4(39)\end{bmatrix} = \tfrac{1}{20}\begin{bmatrix}30\\16\end{bmatrix} = \begin{bmatrix}1.5\\0.8\end{bmatrix}.$$ *Why this step?* Doing the $2\times2$ inversion by hand demystifies "$(\mathbf{X}^\top\mathbf{X})^{-1}$" — it is just this cross-multiply-and-divide-by-determinant. So $\beta_0=1.5,\ \beta_1=0.8$: $\hat y = 1.5 + 0.8x$. ![[deepdives/dd-ai-ml-2.2.02-d3-s01.png]] *Figure — Cell B least-squares fit.* **What to observe:** the blue dots are the four data points; the orange line is the best fit $\hat y=1.5+0.8x$. It passes through *none* of them exactly. Each red dashed segment is a residual $e^{(i)}=y^{(i)}-\hat y^{(i)}$ — the vertical miss, with its signed value labelled. Least-squares chose the one line whose *squared* red lengths add to the smallest possible total; tilting or shifting the line any other way makes that total grow. Notice the reds point up at $x=1,3$ (line below the point) and down at $x=2,4$ (line above), so they balance to zero net miss. > **Verify:** Predict at $x=3$: $1.5+0.8(3)=3.9$; residual $5-3.9=1.1$. Residuals $(2-2.3,\,3-3.1,\,5-3.9,\,4-4.7)=(-0.3,-0.1,1.1,-0.7)$ sum to $0$ ✓ (a guaranteed property of the intercept). SymPy check in VERIFY. --- ## Example 3 — Cell C: mixed signs, holding others constant > [!example] Exam score model > $\hat y = 40 + 8x_1 - 3x_2$, where $x_1=$ study hours, $x_2=$ social-media hours. > Predict a student with $x_1=5, x_2=4$, and one with $x_1=5, x_2=8$. > > **Forecast:** Doubling social media from 4 to 8 — how many points lost? **Step 1 — read the signs.** $\beta_1=+8$ (helps), $\beta_2=-3$ (hurts). Each coefficient is the effect of its feature *holding the other constant* — the defining power of multiple regression. **Step 2 — student A (5, 4):** $40 + 8(5) - 3(4) = 40+40-12 = 68$. *Why this step?* We evaluate the weighted sum for one concrete input so the abstract coefficients become a single tangible score — the model's whole job is turning feature values into one number. **Step 3 — student B (5, 8):** $40 + 8(5) - 3(8) = 40+40-24 = 56$. *Why this step?* We deliberately change *only* $x_2$ (study fixed at 5) so that any difference in the output is attributable solely to social media — this is how you experimentally read off a coefficient's meaning. **Step 4 — isolate the social-media effect.** Because $x_1$ is identical in A and B, the $+8x_1$ and the $+40$ base cancel in the difference. What remains is $\beta_2 \cdot \Delta x_2 = -3\cdot(8-4) = -12$ points. *Why this step?* Subtracting two predictions that differ in one feature is the cleanest visualization of "marginal effect" — it strips away everything held constant. > **Verify:** $68 - 56 = 12$ points lost, exactly $-3\times 4$ ✓. Units: points $=$ (points/hour)$\times$hours ✓. --- ## Example 4 — Cell D: a constant feature (degenerate column) > [!example] A feature that never varies > Suppose every house in your data has exactly 2 bathrooms. You add "bathrooms" as $x_3$. > Data $\mathbf{X}$ column for $x_3$ is $(2,2,2)^\top$. What breaks? > > **Forecast:** Can the model learn a coefficient for a feature that never changes? **Step 1 — spot the twin.** The $x_3$ column $(2,2,2)$ is just $2\times$ the intercept column $(1,1,1)$. Two columns are proportional. **Step 2 — why this is degenerate.** If $x_3 = 2\cdot(\text{ones column})$, then $\beta_0$ and $\beta_3$ can trade off freely: increase $\beta_0$ by $2c$ and decrease $\beta_3$ by $c$ and every prediction is unchanged. Infinitely many equally-good $\boldsymbol\beta$ ⇒ no unique solution. **Step 3 — the diagnostic.** $\mathbf{X}^\top\mathbf{X}$ becomes **singular** (determinant $0$). This is the same disease as Cell E, caused here by a constant column. **Step 4 — the fix.** Drop constant features before fitting; they carry no predictive information. See [[Feature Scaling]] for related preprocessing. > **Verify:** For the $2\times2$ sub-matrix built from columns "ones" and "$2\times$ones", $\det\begin{bmatrix}3 & 6\\ 6 & 12\end{bmatrix}=36-36=0$ ✓ singular. --- ## Example 5 — Cell E: perfect multicollinearity > [!example] One feature is a combo of others > Features: $x_1$ = size (sqft), $x_2$ = size in $\text{m}^2 = x_1/10.764$. Same information, different units. > Three rows of $\mathbf{X}$ (drop intercept for clarity): $(1500, 139.4),(2000,185.8),(1800,167.2)$. > Why can't the Normal Equation solve this? > > **Forecast:** Two columns say the same thing — will $\mathbf{X}^\top\mathbf{X}$ invert? **Step 1 — the exact relation.** $x_2 = k x_1$ with $k = 1/10.764$. One column is a scalar multiple of the other ⇒ ==perfect multicollinearity==. **Step 2 — why $\mathbf{X}^\top\mathbf{X}$ dies.** A linearly dependent column makes $\mathbf{X}$ rank-deficient, so $\mathbf{X}^\top\mathbf{X}$ has determinant $0$ and cannot be inverted. Predictions along the shared direction cannot split credit between $x_1$ and $x_2$. **Step 3 — confirm with the actual data columns.** Take the exact proportional pair $x_2 = \tfrac{1}{10}x_1$ so the arithmetic stays clean (the sqft/m² case is identical in spirit). Using rows $(1500,150),(2000,200),(1800,180)$, the two-column $\mathbf{X}$ produces a $\mathbf{X}^\top\mathbf{X}$ with $\det = 0$ because column 2 is exactly $\tfrac{1}{10}$ of column 1. **Step 4 — the fix menu** (the same remedies flagged in the parent note's mistake box): - Remove one redundant feature. - Use [[Ridge Regression]]: add $\lambda\mathbf{I}$ to make it invertible (see the intuition box in Example 9). - Use [[Gradient Descent]], which still converges to *a* valid solution. > **Verify:** For the actual proportional columns $x_2=\tfrac1{10}x_1$ with rows $(1500,150),(2000,200),(1800,180)$, $\det(\mathbf{X}^\top\mathbf{X})=0$ ✓. See VERIFY. --- ## Example 6 — Cell F: underdetermined ($m < n+1$) > [!example] Two examples, three unknowns > One real feature $x_1$ and an all-zero feature $x_2\equiv 0$, fitting $\hat y = \beta_0 + \beta_1 x_1 + \beta_2 x_2$. Points: $(x_1,x_2,y)=(1,0,2),(2,0,3)$. > How many solutions? > > **Forecast:** $m=2$ examples but $n+1=3$ coefficients, so $m<n+1$ — one answer, none, or infinitely many? **Step 1 — count.** $m=2$ examples, $n+1 = 3$ unknowns $(\beta_0,\beta_1,\beta_2)$. Since $x_2\equiv 0$, it contributes nothing to any prediction yet still owns a free coefficient $\beta_2$. *Why this step?* Comparing rows to unknowns up front tells you immediately whether a unique answer can even exist — fewer equations than unknowns guarantees it cannot. **Step 2 — infinite solutions.** Any value of $\beta_2$ works (it multiplies $0$), so the system is underdetermined: a whole line of valid $\boldsymbol\beta$. *Why this step?* Naming the exact free parameter ($\beta_2$) shows *where* the ambiguity lives, rather than just declaring "no unique solution." **Step 3 — minimum-norm rescue.** The ==pseudoinverse== $(\mathbf{X}^\top\mathbf{X})^+$ picks the solution with smallest $\|\boldsymbol\beta\|$, here setting the useless $\beta_2 = 0$. The genuine line through the two points is $\beta_0=1,\ \beta_1=1$. *Why this step?* When infinitely many answers tie, we still need *one* — the pseudoinverse gives a principled, reproducible choice (the smallest-magnitude coefficients). > **Verify:** $\hat y(1)=1+1=2$ ✓, $\hat y(2)=1+2=3$ ✓, and $\beta_2=0$ minimal ✓. See VERIFY. --- ## Example 7 — Cell G: limiting behaviour > [!example] What dominates as a feature grows huge? > Model $\hat y = 5 + 0.001\,x_1 + 200\,x_2$. As $x_1\to\infty$ with $x_2$ fixed at 1, and as $x_2\to\infty$ with $x_1$ fixed, which term wins? > > **Forecast:** Small coefficient $\times$ huge feature vs large coefficient $\times$ small feature — who dominates? **Step 1 — large $x_1$ limit.** $\hat y \approx 0.001\,x_1$ once $x_1$ is enormous; the $200 x_2$ and $5$ become negligible. A *tiny* coefficient still dominates if its feature grows without bound. *Why this step?* Taking a limit strips away everything that stays bounded, exposing which single term controls the far-field behaviour. **Step 2 — large $x_2$ limit.** $\hat y \approx 200\,x_2$; here the $x_2$ term runs away. *Why this step?* Repeating the limit for the *other* feature shows the "dominant term" depends entirely on which input is growing — there is no universal winner. **Step 3 — the lesson.** Coefficient magnitude alone doesn't tell you influence — it's coefficient $\times$ feature *scale*. This is exactly why we do [[Feature Scaling]] before comparing coefficients or running [[Gradient Descent]]. *Why this step?* Turning the two limits into a rule of thumb is what makes the example transferable to any model you meet later. > **Verify:** At $x_1=10^6,x_2=1$: $\hat y=5+1000+200=1205$, and $0.001x_1=1000$ is $\approx 83\%$ of it ✓. See VERIFY. --- ## Example 8 — Cell H: word problem > [!example] Rideshare fare > A fare model uses distance $d$ (km) and wait time $w$ (min): $\hat y = 2.5 + 1.2\,d + 0.3\,w$ (dollars). > A ride: 8 km, 5 min wait. Then a second ride 12 km, 5 min. What's the per-km rate you're effectively charged? > > **Forecast:** Guess the fare for the 8 km ride before computing. **Step 1 — translate.** Base fare $\beta_0=2.5$, per-km $\beta_1=1.2$, per-minute wait $\beta_2=0.3$. *Why this step?* Naming which real-world number plays which coefficient role is the whole modelling act — get this mapping wrong and every later number is meaningless. **Step 2 — ride A (8, 5):** $2.5 + 1.2(8) + 0.3(5) = 2.5 + 9.6 + 1.5 = 13.6$. *Why this step?* Evaluating the weighted sum turns the abstract rate card into the concrete dollars a rider actually sees. **Step 3 — ride B (12, 5):** $2.5 + 1.2(12) + 0.3(5) = 2.5 + 14.4 + 1.5 = 18.4$. *Why this step?* We hold wait time fixed at 5 min so the *only* thing changing is distance — setting up a clean marginal comparison. **Step 4 — marginal per-km rate.** Same wait ⇒ base and wait terms cancel in the difference: $\Delta \hat y / \Delta d = (18.4-13.6)/(12-8) = 4.8/4 = 1.2$ — exactly $\beta_1$, confirming the coefficient *is* the marginal rate. *Why this step?* Recovering $\beta_1$ from two real fares shows that a coefficient is literally "extra output per one-unit extra input," not an abstract weight. > **Verify:** Fare A $=13.6$, fare B $=18.4$, slope $=1.2 = \beta_1$ ✓. Units: dollars $=$ (dollars/km)(km) ✓. See VERIFY. --- ## Example 9 — Cell I: exam twist (Ridge rescue) Before the rescue, the tool we lean on: > [!definition] What does $\lambda\mathbf{I}$ do? (Ridge, from zero) > $\mathbf{I}$ is the ==identity matrix== — $1$s down the diagonal, $0$s elsewhere; multiplying by it changes nothing. $\lambda$ (lambda) is a small positive number you choose. Adding $\lambda\mathbf{I}$ **bumps every diagonal entry of $\mathbf{X}^\top\mathbf{X}$ up by $\lambda$**. Picture the matrix's "stretch factors" (its eigenvalues): a singular matrix has one that is $0$ (a direction it flattens to nothing, which is why it cannot be undone/inverted). Adding $\lambda$ lifts *every* stretch factor by $\lambda>0$, so none is left at $0$ — the matrix becomes invertible again. As a side effect it gently shrinks the coefficients toward zero, which also fights [[Overfitting]]. Full treatment in [[Ridge Regression]]. > [!example] Make a singular matrix invertible > You have $\mathbf{X}^\top\mathbf{X} = \begin{bmatrix} 2 & 2 \\ 2 & 2 \end{bmatrix}$ (singular — rank 1). Add ridge penalty $\lambda = 1$. Is it now invertible, and what does the fix change? > > **Forecast:** Adding a tiny number to the diagonal — does it break the singularity? **Step 1 — confirm the disease.** $\det\begin{bmatrix}2&2\\2&2\end{bmatrix}=4-4=0$, singular. *Why this step?* A zero determinant is the precise, checkable signature of "cannot be inverted" — we confirm the problem before applying a cure. **Step 2 — apply the [[Ridge Regression]] fix:** replace $\mathbf{X}^\top\mathbf{X}$ with $\mathbf{X}^\top\mathbf{X}+\lambda\mathbf{I}$. $$\begin{bmatrix}2&2\\2&2\end{bmatrix} + 1\cdot\begin{bmatrix}1&0\\0&1\end{bmatrix} = \begin{bmatrix}3&2\\2&3\end{bmatrix}.$$ *Why this step?* Per the definition box, adding $\lambda\mathbf{I}$ lifts the zero stretch factor off the floor, which is exactly the mechanism that restores invertibility. **Step 3 — check.** $\det\begin{bmatrix}3&2\\2&3\end{bmatrix}=9-4=5\neq 0$ ✓ invertible. *Why this step?* Re-computing the determinant *after* the fix proves the cure worked, closing the loop opened in Step 1. > **Verify:** determinant went $0 \to 5$ after ridge ✓. See VERIFY. --- ## Example 10 — Cell J: nearly-singular (ill-conditioned) design > [!example] Almost-collinear features > Two features that are *nearly* (not exactly) proportional: $\mathbf{X}^\top\mathbf{X} = \begin{bmatrix} 1 & 1 \\ 1 & 1.01 \end{bmatrix}$. Is it invertible? Should you trust the answer? > > **Forecast:** The determinant is tiny but nonzero — does the formula still "work"? **Step 1 — check the determinant.** $\det = 1(1.01) - 1(1) = 0.01$. Nonzero, so technically invertible — but *barely*. *Why this step?* The determinant measures how "flat" the matrix is; a value hugging zero warns us the inverse will be delicate before we even compute it. **Step 2 — why "barely" is dangerous.** The inverse divides by the determinant: $(\mathbf{X}^\top\mathbf{X})^{-1} = \tfrac{1}{0.01}\begin{bmatrix}1.01 & -1\\ -1 & 1\end{bmatrix}$, i.e. entries around $100$. A tiny measurement wobble in the data gets *amplified* $100$-fold in $\boldsymbol\beta$. The coefficients become large and unstable. *Why this step?* Writing the inverse out shows *mechanically* how a small denominator explodes every entry — this is what "ill-conditioned" feels like numerically. **Step 3 — how to detect it in practice.** - **Condition number** of $\mathbf{X}^\top\mathbf{X}$ (ratio of largest to smallest eigenvalue) is huge — a red flag if it exceeds ~$10^{3}$–$10^{4}$. - **Correlation matrix** between features shows a pair near $\pm 1$. - **Coefficients flip sign or explode** when you add/remove one data point. *Why this step?* Real datasets never hand you a determinant to eyeball — these are the practical alarms that reveal ill-conditioning in the wild. **Step 4 — the fix.** Same toolkit as Cell E, but now it's *stabilizing* rather than *rescuing*: [[Ridge Regression]] (the $+\lambda\mathbf{I}$ raises the small eigenvalue, shrinking the condition number), drop or combine the near-duplicate feature, or apply [[Feature Scaling]] so features share a comparable range. > **Verify:** $\det\begin{bmatrix}1&1\\1&1.01\end{bmatrix}=0.01\neq0$; after ridge $\lambda=\tfrac1{10}$, $\det\begin{bmatrix}1.1&1\\1&1.11\end{bmatrix}$ jumps well above it. See VERIFY. --- > [!recall]- Which Cell needs which rescue? > Exactly determined means $m$ equals what, not what? ::: $m = n+1$ (rows = coefficients), NOT $m=n$. > Singular $\mathbf{X}^\top\mathbf{X}$ (collinearity / constant feature) — how do you fix it? ::: Drop the redundant feature, use Ridge ($+\lambda\mathbf{I}$), use gradient descent, or the pseudoinverse. > $m < n+1$ (fewer examples than coefficients) — what solution do you get? ::: Infinitely many; the pseudoinverse picks the minimum-norm one. > Nearly-singular (ill-conditioned) — how do you detect it? ::: Huge condition number, feature correlations near ±1, coefficients that explode or flip sign when data changes slightly. > What does adding $\lambda\mathbf{I}$ do to a singular matrix? ::: It lifts every eigenvalue by $\lambda>0$ so none stays at zero, restoring invertibility (and shrinking coefficients). > Coefficient magnitude vs true influence — what else matters? ::: Feature scale; influence is coefficient × feature magnitude, so scale features first. > [!mnemonic] The plug-back oath > **"Never leave a $\boldsymbol\beta$ un-checked."** Every worked answer earns its trust by substituting back into *every* row and confirming units. Related deeper dives: [[Gradient Descent]], [[Ridge Regression]], [[Polynomial Regression]], [[R-squared and Adjusted R-squared]], [[Feature Scaling]], [[Overfitting]].