Exercises — Feature creation and interaction terms

No. A linear model can only build weighted sums of the features you hand it. The product $x_1\cdot x_2$ is multiplicative — it is simply not expressible as $w_1{x}_1+w_2{x}_2+b$ for any fixed weights. Here is the counterexample in full, so nothing is left on faith. Suppose weights $w_1,w_2,b$ did fit exactly. Evaluate at four points:

• $(0,0)$: need $w_1(0)+w_2(0)+b=0\cdot 0=0\Rightarrow b=0$.
• $(1,0)$: need $w_1+b=1\cdot 0=0\Rightarrow w_1=0$.
• $(0,1)$: need $w_2+b=0\cdot 1=0\Rightarrow w_2=0$.
• $(1,1)$: need $w_1+w_2+b=1\cdot 1=1$, but we just forced $w_1=w_2=b=0$, giving $0\ne 1$. Contradiction — no such weights exist. (This exact contradiction is machine-checked in VERIFY.)

Fix: create the interaction feature $x_3=x_1\cdot x_2$. Then $y=w_3{x}_3+b$ with $w_3=1,b=0$ fits exactly. The multiplication was moved out of the model and into the feature.

• (a) scale-invariant → ratio $x_i/x_j$. Dividing cancels the units/scale of the denominator.
• (b) peak/optimum shape → squared term $x_i^2$ (with a negative weight it makes an inverted-U ∩).
• (c) "only matter together" → interaction $x_i\cdot x_j$.

(a) Total terms (minus the constant). Plug in: $\left(\genfrac{}{}{0ex}{}{3+2}{2}\right)-1=\left(\genfrac{}{}{0ex}{}{5}{2}\right)-1=10-1=9$. These 9 are the 3 originals plus 6 higher-order terms: $\underset{\text{3 originals}}{\underbrace{x_1,x_2,x_3}},\underset{\text{6 new}}{\underbrace{x_1^2,x_2^2,x_3^2,x_1{x}_2,x_1{x}_3,x_2{x}_3}}$ (We subtract 1 because the constant term $1$ is redundant with the bias $b$.)

(b) Genuinely new features. You already had $x_1,x_2,x_3$, so subtract those 3: $9-3=6$ new features, namely $x_1^2,x_2^2,x_3^2,x_1{x}_2,x_1{x}_3,x_2{x}_3.$ Watch the wording: the formula $\left(\genfrac{}{}{0ex}{}{n+d}{d}\right)-1$ counts all expansion terms except the constant (here 9); the count of new columns you must add to your existing table is that minus your $n$ originals (here $9-3=6$).

Build the expanded row first: $[T,R,T^2,TR,R^2]=[30,8,900,240,64]$ Now the weighted sum plus bias: $y=2(30)+3(8)+(-0.05)(900)+0.8(240)+(-0.1)(64)+5$ $=60+24-45+192-6.4+5=229.6$ Yield = 229.6. Notice the $-45$ from $T^2$: the negative weight on the squared term is pulling yield down as it gets too hot — that is the inverted-U in action.

Group the terms that multiply $\text{ad}$: $\text{sales}=\text{ad}(50+80\cdot \text{wknd})+200\cdot \text{wknd}+1000.$ The effective ad slope is the thing in the parentheses, $50+80\cdot \text{wknd}$ — it changes with weekend status. That is exactly what an interaction does.

• Weekday ($\text{wknd}=0$): slope $=50+80(0)=50$ per dollar.
• Weekend ($\text{wknd}=1$): slope $=50+80(1)=130$ per dollar.
• (b) Factor $=130/50=2.6\times$.

• (a) Person A: $\frac{50000}{80000}=0.625$.
• (b) Person B: $\frac{27000}{15000}=1.8$. Person B is far riskier: their debt is $1.8\times$ their yearly income, versus A's $0.625\times$.

Why the ratio matters: raw dollars would rank A as "more debt" ($50k>27k$) and possibly flag A as riskier. But the relative burden is what predicts default. Division by income makes the feature scale-invariant — a \100$kearneranda$$15$k earner are compared on the same footing.

Take three points and compute the true $y$:

• $(x_1,x_2)=(1,0)$: true $y=10(1)+5(0)+2(0)=10$, but $x_1{x}_2=0$.
• $(x_1,x_2)=(0,1)$: true $y=0+5+0=5$, but $x_1{x}_2=0$.
• $(x_1,x_2)=(1,1)$: true $y=10+5+2=17$, and $x_1{x}_2=1$.

The interaction-only model gives $y={\beta }_3(x_1{x}_2)+b$. At the first two points $x_1{x}_2=0$, so both must predict the same value $b$. But the truth says one is $10$ and the other is $5$ — impossible to match with a single $b$. The main effects carry information the pure interaction throws away. Always keep main effects when you add their interaction.

Since ${\lambda }_2={\lambda }_1$, the longitude difference is $0$, so ${\sin }^2(0)=0$ and that whole second term vanishes. The formula collapses to a pure latitude arc: $d=2r\arcsin \left(\sqrt{{\sin }^2\left(\frac{{\varphi }_2-{\varphi }_1}{2}\right)}\right)=2r\left|\sin \left(\frac{{\varphi }_2-{\varphi }_1}{2}\right)\right|.$ Convert the $0.1^{\circ }$ latitude difference to radians: $0.1^{\circ }=0.0017453$ rad, half of that $=0.00087266$ rad. $\sin$ of a tiny angle $\approx$ the angle, so $d\approx 2(6371)(0.00087266)\approx 11.12\text{km}.$ Why raw columns fail: fare depends on distance travelled, a nonlinear combination of four coordinates. A linear model on $[{\varphi }_1,{\lambda }_1,{\varphi }_2,{\lambda }_2]$ can only weight each separately — it cannot compute a square root of squared differences. The engineered distance feature is the physics the model needs.

A defensible set (many valid answers — the reasoning is graded). Crucially, none of these use price, because feeding the target into a feature is data leakage that makes the model look perfect in training and useless in production:

1. Ratio — sqft_per_bedroom = sqft / (bedrooms + 1): exposes how spacious each room is, scale-invariant to house size. (The $+1$ guards against dividing by zero for a listing recorded with $0$ bedrooms.)
2. Difference — age = current_year − year_built: raw year_built is a meaningless absolute number; the age (a difference of two years) is what depreciates value. This is the difference operation from the taxonomy.
3. Interaction — sqft × bedrooms: extra floor area is worth more when there are more rooms to place it in — a joint effect a plain sum can't express.
4. Polynomial — age²: value often drops fast then levels off (or historic homes regain value) — an inverted-U/curved effect needs the squared term.

Blow-up control: apply Regularization (L1, L2) — specifically L1 to zero-out useless engineered columns — and validate the whole pipeline with Cross-Validation so you never judge features on the training set alone. For a tree-based baseline, Decision Trees and Random Forests can find some interactions automatically, giving you a sanity check on which engineered terms actually help.

Factor out $x_1$: $y=x_1({\beta }_1+{\beta }_3{x}_2)+{\beta }_2{x}_2+b.$ The coefficient multiplying $x_1$ is $({\beta }_1+{\beta }_3{x}_2)$. Because $x_2$ sits inside it, the slope of $y$ in the $x_1$ direction is ${\beta }_1+{\beta }_3{x}_2$ — it moves as $x_2$ moves. (Formally, the partial derivative $\partial y/\partial x_1={\beta }_1+{\beta }_3{x}_2$.)

Moderation: when the effect of one variable on the outcome depends on the level of a second variable, the second variable is said to moderate the first — and the interaction term ${\beta }_3{x}_1{x}_2$ is exactly how a linear model encodes it.