Feature creation and interaction terms

What Are We Really Doing?

The goal: increase the expressive power of your feature space so linear models can capture nonlinear relationships, and complex models can learn faster with less data.

Why Feature Creation Matters

The Representation Bottleneck: Most ML algorithms learn coefficients/weights on features. If your features don't encode the right relationships, no amount of training will help.

Figure — Feature creation and interaction terms

Mathematical Reality: A linear model $y = w_1x_1 + w_2x_2 + b$ can NEVER learn $y = x_1 \cdot x_2$ no matter how much data you have—the multiplicative relationship isn't in the hypothesis space. But if you create $x_3 = x_1 \cdot x_2$ , suddenly $y = w_3x_3 + b$ works perfectly.

Types of Feature Creation

1. Polynomial Features

Derivation from scratch:

Why? Real-world relationships are rarely linear. Physics: $E = \frac{1}{2}mv^2$ (kinetic energy), $A = \pi r^2$ (area).
How many features? For $n$ features at degree $d$ :

\text{New features} = \binom{n + d}{d} - 1

For $n=2, d=2$ : $\binom{4}{2} - 1 = 6 - 1 = 5$ new features: $x_1^2, x_1x_2, x_2^2, x_1, x_2$
Why subtract 1? The constant term 1 is redundant with the bias.

But yield might peak at moderate temperature (too hot/cold is bad) and depend on the combination of temperature and water availability.

Solution: Create degree-2 polynomial features:

[T, R, T^2, TR, R^2]

Model becomes:

\text{yield} = w_1T + w_2R + w_3T^2 + w_4(TR) + w_5R^2 + b

Why this works:

$T^2$ with negative $w_3$ captures the inverted-U shape (optimal temperature)
$TR$ captures interaction: rain helps more when it's warm
$R^2$ might capture diminishing returns (too much rain floods crops)

Concrete numbers:

Original: $T=25°C, R=10\text{mm}$ → features $[25, 10]$
Expanded: $[25, 10, 625, 250, 100]$
If learned weights are $[2, 3, -0.05, 0.8, -0.1]$ :

\text{yield} = 2(25) + 3(10) - 0.05(625) + 0.8(250) - 0.1(100) + 5 = 50 + 30 - 31.25 + 200 - 10 + 5 = 243.75

2. Interaction Terms

Why multiplication? It encodes joint effect where the impact of one feature depends on the value of another.

Mathematical intuition: Consider a model with interaction:

y = \beta_1 x_1 + \beta_2 x_2 + \beta_3 (x_1 \cdot x_2) + \epsilon

Rearrange:

y = x_1(\beta_1 + \beta_3 x_2) + \beta_2 x_2 + \epsilon

The effective coefficient of $x_1$ is $(\beta_1 + \beta_3 x_2)$ —it changes with $x_2$ ! This is moderation/synergy.

Naive model:

\text{sales} = 50 \cdot \text{ad\_spend} + 200 \cdot \text{is\_weekend} + 1000

Issue: This says ads have the same $50/\backslash$ impact on weekdays and weekends. But weekend shopers might be more responsive!

Add interaction:

\text{interaction} = \text{ad\_spend} \times \text{is\_weekend}

New model:

\text{sales} = 50 \cdot \text{ad\_spend} + 200 \cdot \text{is\_weekend} + 80 \cdot (\text{ad\_spend} \times \text{is\_weekend}) + 1000

Interpretation:

Weekday ( $\text{is\_weekend}=0$ $is_weekend = 0$ ): sales $= 50 \cdot \text{ad\_spend} + 1000$ $= 50 \cdot ad_spend + 1000$
- Each ad dollar → +$50 sales
Weekend ( $\text{is\_weekend}=1$ $is_weekend = 1$ ): sales $= 50 \cdot \text{ad\_spend} + 200 + 80 \cdot \text{ad\_spend} + 1000$ $= 50 \cdot ad_spend + 200 + 80 \cdot ad_spend + 1000$
- Simplify: sales $= 130 \cdot \text{ad\_spend} + 1200$
- Each ad dollar → +$130 sales (2.6× more effective!)

Why this step? The interaction term $80 \cdot \text{ad\_spend} \cdot \text{is\_weekend}$ only activates on weekends, adding extra slope to the ad_spend effect.

3. Ratio and Difference Features

When to use:

Ratios: Normalize by scale (price-to-earnings ratio, clicks-per-impression)
Differences: Capture change (temperature delta, account balance change)
Rates: Efficiency metrics (revenue per employee, error per request)

A model might learn: high debt → high risk. But $50k debt on$ 80k income is different from $50k on$ 30k income!

Create:

\text{debt\_to\_income\_ratio} = \frac{\text{total\_debt}}{\text{annual\_income}} = \frac{50000}{80000} = 0.625

Now the model sees relative burden, not absolute numbers. Someone with 0.625 ratio is safer than someone with 1.8 ratio, regardless of actual dollar amounts.

Why this step? Division normalizes by the denominator, creating a scale-invariant feature.

4. Domain-Specific Features

Create:

price_lag1 = yesterday's price
price_diff = today's price - yesterday's price (daily change)
rolling_mean_7 = average price over last 7 days (trend)
volatility = std(last 7 days) (risk measure)

Why? These capture temporal patterns: trends, mean-reversion, volatility clustering.

Create Haversine distance:

d = 2r \arcsin\left(\sqrt{\sin^2\left(\frac{\phi_2-\phi_1}{2}\right) + \cos(\phi_1)\cos(\phi_2)\sin^2\left(\frac{\lambda_2-\lambda_1}{2}\right)}\right)

where $\phi$ = latitude, $\lambda$ = longitude, $r$ = Earth radius.

Why? Distance is the primary driver of fare. Raw coordinates don't directly encode this.

Common Mistakes

Common mistake Mistake 1: Creating Too Many Features (Combinatorial Explosion)

Wrong idea: "More features = more information, so create ALL possible interactions!" Why it feels right: Polynomial degree 3, 4, 5... must capture even more complex patterns.

The truth: With $n=10$ features at degree 3, you get $\binom{13}{3} = 286$ features. Degree 4 → 1001 features. You've created:

Overfitting risk: Model memorizes noise in the polynomial combinations
Computational cost: Training time $O(n^2)$ or $O(n^3)$ for many algorithms
Multicollinearity: $x_1^2$ and $x_1\cdot x_2$ can be highly correlated Fix:

Use domain knowledge to create targeted interactions
Apply regularization (L1/L2) to automatically select useful features
Use feature selection methods (mutual information, recursive elimination)
Start with degree2, evaluate, then cautiously go to degree 3 if needed

Common mistake Mistake 2: Interaction Without Main Effects

Wrong idea: Include $x_1 \cdot x_2$ but drop $x_1$ and $x_2$ from the model.

Why it feels right: The interaction term captures their relationship, so the individual terms are redundant.

The truth: Consider the true model $y = 10x_1 + 5x_2 + 2(x_1 x_2)$ . If you only fit $y = \beta(x_1 x_2)$ :

When $x_2 = 0$ : true effect is $10x_1$ , but your model predicts 0 (the interaction vanishes!)
You're forcing the model to explain main effects through interactions, which distorts estimates

Fix: Hierarchical principle: Always include lower-order terms when including higher-order terms. If $x_1 x_2$ is in the model, $x_1$ and $x_2$ must be too. This gives the model flexibility to separate additive and multiplicative effects.

Common mistake Mistake 3: Creating Features After Train-Test Split

Wrong idea: Split data first, then compute mean(target) by category on training set and apply to test set.

Why it feels right: I'm only using training data to compute the mean, so no leakage.

The truth: Leakage can be subtle. Example: creating user_avg_purchase as the mean of that user's historical purchases:

If you compute this on the full dataset before splitting, future test examples' purchase amounts influenced the feature value
Even if you compute only on train: if the same user appears in both train and test, and you're predicting user-level behavior, you've leaked test information

Fix:

Split data first (or use time-based split for temporal data)
Compute aggregations/transformations only on training data
Apply the same transformation to test using training statistics (e.g., training mean, not test mean)
For time-series: use only past data to predict future (lag features, rolling windows with proper cutoffs)

Common mistake Mistake 4: Forgetting to Scale After Feature Creation

Wrong idea: Create x1_squared and x1_x2, then train without normalizing.

Why it feels right: The model will learn appropriate weights for each feature. The truth: If $x_1 \in [0, 100]$ , then $x_1^2 \in [0, 10000]$ . For gradient-based algorithms:

Weight updates for $x_1^2$ will be 100× larger in magnitude than for $x_1$
This creates ill-conditioned optimization landscapes (long, narrow valleys)
Convergence is slow and unstable

Fix: After creating polynomial/interaction features, apply standardization:

x'_j = \frac{x_j - \mu_j}{\sigma_j}

This puts all features on comparable scales. Use sklearn.preprocessing.StandardScaler after PolynomialFeatures.

[!recall]- Explain It to a 12-Year-Old

Imagine you're trying to predict how fun a video game is using just two numbers: graphics quality (1-10) and story quality (1-10).

If you just add them up ( $\text{fun} = \text{graphics} + \text{story}$ ), you're saying a game with great graphics but no story (10 + 2) is just as fun as a game with okay graphics and story (6 + 6). But that's wrong!

Feature creation is like creating a NEW number that captures what you really care about:

$\text{graphics} \times \text{story}$ = a "synergy score." If BOTH are good, this number is huge (10 × 10 = 100). If one is terrible, even if the other is great, this is small (10 × 2 = 20). This interaction captures that both need to be good together.
$\text{graphics}^2$ captures that going from 9→10 in graphics quality might matter MORE than going from 2→3 (diminishing returns at the low end, excitement at the high end).

By creating these new numbers from the old ones, you help the computer see patterns it couldn't see before. It's like giving it new "senses" to understand the world.

[!mnemonic] PRIME Features

Polynomial (powers for curves) Ratios (normalize by scale) Interactions (joint effects, products) Mathematical (logs, sqrt, domain transforms) Engineered (lags, rolling, domain-specific)

Practical Implementation

import numpy as np
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import Ridge
 
# Sample data: predicting sales from ad_spend and website_traffic
X_train = np.array([[100, 500], [150, 800], [200, 1200], [80, 400]])
y_train = np.array([5000, 7500, 12000, 3800])
 
# Create polynomial features with interactions
poly = PolynomialFeatures(degree=2, include_bias=False, interaction_only=False)
X_poly = poly.fit_transform(X_train)
 
# Features are now: [ad, traffic, ad^2, ad*traffic, traffic^2]
print("Original shape:", X_train.shape)  # (4, 2)
print("Polynomial shape:", X_poly.shape)  # (4, 5)
print("Feature names:", poly.get_feature_names_out(['ad_spend', 'traffic']))
# ['ad_spend', 'traffic', 'ad_spend^2', 'ad_spend*traffic', 'traffic^2']
 
# Always scale after creating features!
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_poly)
 
# Train model
model = Ridge(alpha=1.0)
model.fit(X_scaled, y_train)
 
# Manual interaction creation example
X_manual = X_train.copy()
interaction = X_train[:, 0] * X_train[:, 1]  # ad_spend * traffic
X_with_interaction = np.column_stack([X_train, interaction])
print("With manual interaction:", X_with_interaction.shape)  # (4, 3)

When to Use Feature Creation

Use polynomial features when:

You have small feature sets (< 20 features) to avoid explosion
Nonlinear relationships are suspected (physics, economics, biology)
Using linear models that need help capturing curves
You have sufficient data ( $n \gg$ new feature count) to avoid overfitting

Use interaction terms when:

Domain knowledge suggests synergy (medication + diet, temperature + humidity)
A/B testing reveals different effects for different segments
Exploratory data analysis shows crossing effects in plots
Modeling moderating variables (the effect of X on Y depends on Z)

Use domain features when:

You have temporal data (lags, rolling stats, seasonality)
You have spatial data (distances, coordinates, neighborhoods)
You have text/categorical data (encoding tricks, embedings)
There are known physical laws to encode (conservation laws, rate equations)

Avoid feature creation when:

Using tree-based models (XGBoost, Random Forest) for polynomial features—they capture interactions and nonlinearity naturally through splits
You have too many features already (high-dimensional, genomics)—add more features cautiously
Your model is already overfitting—simplify first, add features later
You lack domain expertise and are creating features blindly—this is feature engineering roulette

Connections

Exploratory Data Analysis - visualize relationships to inspire feature ideas
Feature Scaling and Normalization - always scale after creating polynomial/interaction features
Regularization (L1, L2) - control overfitting when you've added many features
Feature Selection Methods - prune irrelevant created features
Overfitting and Underfitting - more features increases overfitting risk
Linear Regression - polynomial features turn linear models into nonlinear ones
Decision Trees and Random Forests - capture interactions naturally, less need for manual feature engineering
Cross-Validation - evaluate whether new features actually improve generalization

#flashcards/ai-ml

What is the primary goal of feature creation in machine learning? :: To increase the expressive power of the feature space by transforming and combining existing variables, allowing models to capture relationships that aren't present in the raw data.

Why can't a linear model $y = w_1x_1 + w_2x_2 + b$ ever learn a multiplicative relationship $y = x_1 \cdot x_2$ ? :: Because multiplication is not in the hypothesis space of linear combinations. The model can only learn additive weighted sums, not products of features. Creating the interaction term $x_3 = x_1 \cdot x_2$ explicitly adds this relationship.

How many features result from applying degree-2 polynomial expansion to 3 original features?

\binom{3+2}{2} - 1 = \binom{5}{2} - 1 = 10 - 1 = 9

features (including the 3 original linear terms,3 squared terms, and 3 pairwise interaction terms).

What does an interaction term

x_1 \times x_2

capture that individual features

x_1

and

x_2

cannot?

It captures how the effect of one feature depends on the value of the other (moderation/synergy). The coefficient of

x_1

effectively becomes

(\beta_1 + \beta_3 x_2)

, varying with

x_2

Why should you create ratio features like debt-to-income instead of using absolute values?

Ratios create scale-invariant features that capture relative relationships. $50k debt means different things for

30k vs \

80k income—the ratio normalizes by the denominator to encode relative burden.

State the hierarchical principle for interaction terms.

Always include lower-order terms when including higher-order terms. If

x_1 x_2

is in your model, both

x_1

and

x_2

must also be included to allow the model to separate additive and multiplicative effects.

Why must you scale features after creating polynomial or interaction terms?

Because squared and interaction features have much larger magnitudes (e.g.,

x_1^2 \in [0, 10000]

x_1 \in [0, 100]

), creating ill-conditioned optimization landscapes. Standardization puts all features on comparable scales for stable gradient descent.

When should you avoid creating polynomial features for tree-based models like Random Forest?

Tree-based models naturally capture nonlinearity and interactions through recursive splits on single features. Adding polynomial features is redundant and increases dimensionality without benefit, potentially slowing training.

What is the correct order: split data then create features, or create features then split data?

Split data FIRST, then create features using only training data statistics. Computing aggregations on the full dataset before splitting causes target leakage—test information influences feature values.

For time-series prediction, why create lag features like price_lag_1 and rolling averages?

They capture temporal dependencies: lag features encode autocorrelation (today depends on yesterday), rolling averages capture trends, and rolling standard deviations capture volatility. Raw current values lack this temporal context.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Feature creation ka matlab hai ke hum apne existing data se naye variables banate hain jo hidden patterns ko expose karte hain. Socho agar tumhe ghar ki price predict karni hai sirf square footage aur number of bedrooms se—ye pura picture nahi dete. Agar tum create karo "price per square foot" (price / sqft), tab suddenly model ko efficiency ka idea mil gaya. Ye transformation data ko model ke liye "digestible" bana deta hai.

Interaction terms bohot powerful hain kyunki ye capture karte hain ke do features sath milkar kaise kaam karte hain. Example: temperature akela aur sunny weather akela ice cream sales ko fully explain nahi kar sakte, lekin "temperature × is_sunny" ye compound effect capture kar leta hai. Jab dono high hain, tab interaction term ka value explode hota hai, jo model ko bata hai ke ye combination special hai.

Domain knowledge inject karna bhi zaroori hai. Agar tum time-series data pe kaam kar rahe ho, tab "lag features" (kal ki value) aur "rolling averages" (pichle 7 din ka mean) banao. Agar geospatial data hai, tab latitude-longitude se Haversine distance calculate karo. Physics ya business rules ko explicitly features mein encode karne se model ko seekhna easy ho jata hai—warna wo raw numbers se ye relationships discover karne mein struggle karega. Feature engineering wahi hai jo average models ko great banata hai, kyunki sahi features ke sath even simple linear regression bhi nonlinear patterns capture kar sakti hai!

What Are We Really Doing?

Why Feature Creation Matters

Types of Feature Creation

1. Polynomial Features

2. Interaction Terms

3. Ratio and Difference Features

4. Domain-Specific Features

Common Mistakes

[!recall]- Explain It to a 12-Year-Old

[!mnemonic] PRIME Features

Practical Implementation

When to Use Feature Creation

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Data Preprocessing & Feature Engineering

Connections