Feature creation and interaction terms: Visual walkthrough

Everything here builds on the parent note Feature creation and interaction terms. If a symbol shows up, we define it first.

Step 1 — One feature, one straight line

WHAT. We start with the simplest possible predictor. We have one input number, call it $x_{1}$ (say ad spend in dollars), and one output we want to guess, call it $y$ (say sales). We propose:

y = starting height β_{0} + slope β_{1} \cdot the input x_{1}

$β_{0}$ ("beta-zero") is the intercept — where the line sits when $x_{1} = 0$ . It is the sales you'd make spending nothing on ads.
$β_{1}$ ("beta-one") is the slope — how many extra dollars of sales you get for each extra dollar of ad spend. Slope means "rise over run": rise in $y$ divided by run in $x_{1}$ .

WHY. Before adding anything fancy, we need to see what "linear" even buys us. A line has exactly one steepness, and that steepness is the same everywhere.

PICTURE. In the figure the blue line rises at a fixed rate. Pick any two points one unit apart along $x_{1}$ — the jump in $y$ (the red rise) is identical no matter where you stand. That "same jump everywhere" is the whole personality of a line.

Step 2 — Two features: the flat sheet

WHAT. Add a second input $x_{2}$ . To keep it concrete, let $x_{2}$ be is_weekend, a number that is either $0$ (weekday) or $1$ (weekend). Our model grows one term:

y = β_{0} + ad-spend effect β_{1} x_{1} + weekend effect β_{2} x_{2}

$β_{2}$ is how much sales shift just for it being the weekend, before any ads.

WHY. Two inputs means $y$ now lives above a plane — we plot $x_{1}$ on one floor axis, $x_{2}$ on the other, and $y$ as height. A model with only $β_{1} x_{1} + β_{2} x_{2}$ is a flat tilted sheet floating over that floor.

PICTURE. Look at the two blue lines in the figure — one is the weekday slice ( $x_{2} = 0$ ), one the weekend slice ( $x_{2} = 1$ ). They are parallel. The weekend line is lifted by $β_{2}$ , but it climbs at the same slope $β_{1}$ . That parallelness is the flatness of the sheet: sliding along $x_{2}$ only raises or lowers the line, never tilts it.

Step 3 — The question a flat sheet cannot answer

WHAT. Ask a sharper question: does each ad dollar do more on weekends? In symbols, we want the slope with respect to $x_{1}$ to depend on $x_{2}$ . But on the flat sheet the slope in the $x_{1}$ direction is:

\frac{\partial y}{\partial x _{1}} = β_{1} (a plain constant — no x_{2} in sight)

The symbol $\frac{\partial y}{\partial x _{1}}$ ("partial derivative") just means: hold $x_{2}$ fixed and measure how fast $y$ climbs as $x_{1}$ grows — the slope of the $x_{1}$ -slice. We use a derivative here specifically because "how the effect of one feature changes" is a slope-of-a-slope question, and derivatives are the tool that reads off slopes.

WHY. The result $β_{1}$ has no $x_{2}$ in it. Mathematically, that is a proof the flat sheet can never let weekends change the ad slope. We are stuck.

PICTURE. The figure overlays the two parallel slices again and marks their slopes with a protractor angle. Both angles are equal. There is no room in $β_{1} x_{1} + β_{2} x_{2}$ to make them differ.

Step 4 — Introduce the product term $x_{1} x_{2}$

WHAT. Create a brand-new feature by multiplying the two inputs: $x_{1} x_{2}$ . Give it its own weight $β_{3}$ and add it in:

y = β_{0} + β_{1} x_{1} + β_{2} x_{2} + interaction term β_{3} (x_{1} x_{2})

$x_{1} x_{2}$ is literally ad-spend times the 0/1 weekend flag.
$β_{3}$ is the weight the model learns for that product.

WHY multiplication and not, say, addition? Because a product is the smallest thing that is zero when either input is zero and grows with both together. On weekdays $x_{2} = 0$ , so $x_{1} x_{2} = 0$ and the term switches off. On weekends $x_{2} = 1$ , so $x_{1} x_{2} = x_{1}$ and the term switches on. A sum could never toggle like that; a product is the natural "both-at-once" gate.

PICTURE. The figure shows the value of $x_{1} x_{2}$ across the floor. It stays a flat zero along the entire weekday edge, then ramps up along the weekend edge — a triangular ramp pinned to the $x_{2} = 1$ side.

Step 5 — Watch the slope become a variable

WHAT. Recompute the ad-spend slope, now with the interaction present. Group the $x_{1}$ terms:

y = effective slope of x_{1} (β_{1} + β_{3} x_{2}) x_{1} + β_{2} x_{2} + β_{0}

WHY. The quantity in front of $x_{1}$ is now $(β_{1} + β_{3} x_{2})$ — it contains $x_{2}$ . That is exactly the thing Step 3 said was impossible on the flat sheet. Take the partial derivative again to make it airtight:

\frac{\partial y}{\partial x _{1}} = β_{1} + β_{3} x_{2}

On a weekday ( $x_{2} = 0$ ): slope $= β_{1}$ .
On a weekend ( $x_{2} = 1$ ): slope $= β_{1} + β_{3}$ .

The two slices no longer have to be parallel — $β_{3}$ is the difference in their slopes.

PICTURE. The two blue lines now fan out from a shared starting point: same intercept behaviour, different steepness. The orange gap between them widens as $x_{1}$ grows — that widening is $β_{3} x_{1}$ .

Step 6 — When $x_{2}$ is continuous, not just 0/1

WHAT. So far $x_{2}$ has only been a switch ( $0$ or $1$ ). But the exact same algebra works when $x_{2}$ is a continuous number — say $x_{2}$ is now temperature instead of a weekend flag. The model is unchanged:

y = β_{0} + β_{1} x_{1} + β_{2} x_{2} + β_{3} (x_{1} x_{2}), \frac{\partial y}{\partial x _{1}} = β_{1} + β_{3} x_{2}

WHY. With a binary $x_{2}$ there were exactly two slopes: $β_{1}$ (off) and $β_{1} + β_{3}$ (on) — two lines. With a continuous $x_{2}$ the slope $β_{1} + β_{3} x_{2}$ slides smoothly as $x_{2}$ moves: every temperature gets its own ad-slope. So instead of two lines fanning apart, you get a whole continuous fan of lines — one per value of $x_{2}$ — sweeping steeper and steeper as $x_{2}$ grows.

PICTURE. The figure draws several slices for $x_{2} = 0, 1, 2, 3, 4$ . Notice they are no longer two lines but a graded fan: each step up in $x_{2}$ tilts the line a little more by exactly $β_{3}$ . The switch has become a dimmer with a continuous dial.

Step 7 — The sheet is now a twisted surface

WHAT. Put both inputs back into 3-D at once. Instead of a flat plane, $y$ now sits on a surface that we call bilinear.

WHY. This twisted surface is the geometric meaning of "interaction." The corner over (high $x_{1}$ , high $x_{2}$ ) is lifted extra high by $β_{3} x_{1} x_{2}$ , pulling one corner up and creating a gentle saddle-like twist. That corner lift is the synergy we wanted to model.

PICTURE. The mesh in the figure is a straight line along the front edge and along the side edge, but the far corner warps upward. Trace any single edge: it is dead straight (Step 1's promise survives along one direction), yet the whole sheet is no longer flat.

Step 8 — Edge, degenerate, and practical cases (never leave the reader stranded)

WHAT & WHY & PICTURE, case by case — the figure shows four in one panel:

$β_{3} = 0$ . The product term vanishes; slices go parallel again; we collapse back to the flat sheet of Step 2. Interaction is a strict generalization — the flat model is the special case $β_{3} = 0$ .
$x_{2} = 0$ everywhere (feature never "on"). Then $x_{1} x_{2} \equiv 0$ , the whole column is zeros, and $β_{3}$ is unlearnable — the fit has no information to set it. This is why a constant/degenerate feature makes interactions useless.
Interaction without main effects. Drop $β_{1} x_{1}$ and $β_{2} x_{2}$ , keep only $β_{3} x_{1} x_{2}$ . Now you can no longer describe an ad effect that exists on its own — the surface is pinned flat along both axes. (This is Mistake 2 from the parent note, shown geometrically.)
$β_{3}$ negative. The far corner dips down instead of up — antagonism, not synergy. Ads could pay less on weekends. Same machinery, opposite twist.

Common mistake Practical pitfall — multicollinearity, and why we center

When $x_{1}$ and $x_{2}$ are both large positive numbers, the product $x_{1} x_{2}$ is strongly correlated with $x_{1}$ and with $x_{2}$ themselves. This multicollinearity makes the estimated weights $β_{1}, β_{2}, β_{3}$ wobbly and hard to interpret (a small data change swings them wildly). Standard fix: center each feature first — subtract its mean so it hovers around zero: $\tilde{x} = x - \overset{x}{ˉ}$ . Then build the interaction from the centered versions, $\tilde{x}_{1} \tilde{x}_{2}$ . Centering shrinks the correlation between the product and its parents dramatically, and it makes each main-effect $β$ read as "the slope at the average of the other feature." See Feature Scaling and Normalization and Regularization (L1, L2) for the tools that further tame this.

The one-picture summary

How to read the figure below. Two panels share the same axes: the horizontal axis is $x_{1}$ (ad spend) and the vertical axis is $y$ (sales). In each panel, the blue line is the weekday slice ( $x_{2} = 0$ ) and the orange line is the weekend slice ( $x_{2} = 1$ ).

Left panel — the flat model. Blue and orange are parallel: the ad-slope is the constant $β_{1}$ no matter the day. Weekend is only lifted by $β_{2}$ .
Right panel — after adding $x_{1} x_{2}$ . The orange line fans away from the blue one: its slope is now $β_{1} + β_{3}$ , so the gap between them grows with $x_{1}$ . That growing gap is the interaction — the constant slope has become a variable slope.

The single takeaway: add the product $x_{1} x_{2}$ and the constant slope $β_{1}$ becomes the variable slope $β_{1} + β_{3} x_{2}$ — parallel lines become fanned lines, a flat sheet becomes a twisted one.

Recall Feynman retelling — say it to a 12-year-old

Imagine a ramp that turns dollars of advertising into dollars of sales. On a normal day the ramp has a certain steepness — every ad dollar gives the same reward. Now you notice weekends feel different: the same dollar seems to work harder. A plain model can only lift the whole ramp up on weekends; it's stuck at the same steepness because its steepness is a fixed number with no "weekend" inside it. So we invent one new number: ad-spend multiplied by the weekend flag. On weekdays that number is zero (the weekend flag is 0), so nothing changes. On weekends it becomes just the ad-spend again, and it adds extra steepness to the ramp — an amount we call $β_{3}$ . Now the ramp can be gentle on weekdays and steep on weekends. In pictures the two ramps stop being parallel and fan apart, and the flat sheet over the whole map develops a twist at the busy corner. If the second feature is a smooth number like temperature instead of a yes/no flag, you don't get two ramps — you get a whole sliding fan of ramps, one steepness per temperature. If $β_{3}$ is zero we're back to boring parallel ramps; if it's negative the weekend ramp is shallower. One warning from the real world: when both inputs are big, the product looks a lot like the inputs themselves, which confuses the fit, so we usually center the features (shift them to sit around zero) before multiplying. That single multiply-two-things-together trick is the entire idea of an interaction term.

Recall Quick self-check

What is the effective slope of $x_{1}$ once the interaction is added? ::: $β_{1} + β_{3} x_{2}$ — it now depends on $x_{2}$ . Why a product and not a sum for interactions? ::: A product is zero when either input is zero and grows with both together, so it acts like a switch $\times$ dimmer; a sum can only shift, never toggle a slope. What does $β_{3} = 0$ mean geometrically? ::: The twist disappears; the slices go parallel; you recover the plain flat-sheet linear model. What changes when $x_{2}$ is continuous instead of 0/1? ::: You get a smooth fan of lines — one slope $β_{1} + β_{3} x_{2}$ per value of $x_{2}$ — instead of just two lines. Why do we usually center features before making an interaction? ::: To reduce multicollinearity between the product and its parent features, stabilising and clarifying the fitted weights.

See also: Linear Regression (the flat sheet), Overfitting and Underfitting (why not add every product), Regularization (L1, L2) and Feature Selection Methods (taming too many interactions), Feature Scaling and Normalization (centering before interacting), Exploratory Data Analysis (spotting where a twist is needed), Decision Trees and Random Forests (models that find interactions automatically), Cross-Validation (checking a new interaction actually helps).