Feature creation and interaction terms — Hinglish notes

2.1.10 · AI-ML › Data Preprocessing & Feature Engineering

Intuition The Core Idea

Raw data rarely perfect form mein aata hai learning ke liye. Feature creation ek art hai jisme existing variables ko transform aur combine kiya jaata hai taaki woh hidden patterns expose ho sakein jo simple models detect nahi kar paate. Aise socho: agar tum sirf "square footage" aur "number of bedrooms" se house prices predict kar rahe ho, toh tum rate information miss kar rahe ho—price per square foot absolute size se zyada matter karta hai. price_per_sqft = price / sqft create karna is relationship ko explicitly reveal kar deta hai.

Interaction terms capture karte hain ki do features milke kaise kaam karte hain — jo inhe alag-alag examine karne se pata nahi chalta. Temperature akela ice cream sales ke baare mein zyada nahi batata, aur sunny weather bhi nahi—lekin temperature × is_sunny compounding effect ko capture karta hai.

What Are We Really Doing?

Goal yeh hai: apne feature space ki expressive power badhaana taaki linear models nonlinear relationships capture kar sakein, aur complex models kam data mein jaldi seekh sakein.

Why Feature Creation Matters

The Representation Bottleneck: Zyaatar ML algorithms features par coefficients/weights seekhte hain. Agar tumhare features sahi relationships encode nahi karte, toh kitni bhi training help nahi karegi.

Figure — Feature creation and interaction terms

Mathematical Reality: Ek linear model $y = w_{1} x_{1} + w_{2} x_{2} + b$ kabhi bhi $y = x_{1} \cdot x_{2}$ nahi seekh sakta, chahe kitna bhi data ho—multiplicative relationship hypothesis space mein hai hi nahi. Lekin agar tum $x_{3} = x_{1} \cdot x_{2}$ create karo, toh achanak $y = w_{3} x_{3} + b$ perfectly kaam karta hai.

Types of Feature Creation

1. Polynomial Features

Scratch se derivation:

Kyun? Real-world relationships rarely linear hote hain. Physics: $E = \frac{1}{2} m v^{2}$ (kinetic energy), $A = π r^{2}$ (area).
Kitne features? $n$ features ke liye degree $d$ par:

New features = (d n + d) - 1

$n = 2, d = 2$ ke liye: $(2 4) - 1 = 6 - 1 = 5$ new features: $x_{1}^{2}, x_{1} x_{2}, x_{2}^{2}, x_{1}, x_{2}$
1 kyun subtract karte hain? Constant term 1 bias ke saath redundant hai.

Lekin yield moderate temperature par peak kar sakta hai (bahut zyada garam/thanda bura hai) aur temperature aur water availability ke combination par depend kar sakta hai.

Solution: Degree-2 polynomial features create karo:

[T, R, T^{2}, T R, R^{2}]

Model ban jaata hai:

yield = w_{1} T + w_{2} R + w_{3} T^{2} + w_{4} (T R) + w_{5} R^{2} + b

Yeh kyun kaam karta hai:

Negative $w_{3}$ ke saath $T^{2}$ inverted-U shape capture karta hai (optimal temperature)
$T R$ interaction capture karta hai: rain tab zyada help karta hai jab warm ho
$R^{2}$ diminishing returns capture kar sakta hai (bahut zyada rain crops ko flood kar deta hai)

Concrete numbers:

Original: $T = 25° C, R = 10 mm$ → features $[25, 10]$
Expanded: $[25, 10, 625, 250, 100]$
Agar learned weights hain $[2, 3, - 0.05, 0.8, - 0.1]$ :

yield = 2 (25) + 3 (10) - 0.05 (625) + 0.8 (250) - 0.1 (100) + 5 = 50 + 30 - 31.25 + 200 - 10 + 5 = 243.75

2. Interaction Terms

Multiplication kyun? Yeh joint effect encode karta hai jahan ek feature ka impact doosre feature ki value par depend karta hai.

Mathematical intuition: Interaction ke saath ek model consider karo:

y = β_{1} x_{1} + β_{2} x_{2} + β_{3} (x_{1} \cdot x_{2}) + ϵ

Rearrange karo:

y = x_{1} (β_{1} + β_{3} x_{2}) + β_{2} x_{2} + ϵ

$x_{1}$ ka effective coefficient hai $(β_{1} + β_{3} x_{2})$ —yeh $x_{2}$ ke saath change karta hai! Yahi moderation/synergy hai.

Naive model:

sales = 50 \cdot ad_spend + 200 \cdot is_weekend + 1000

Issue: Yeh kehta hai ki ads ka weekdays aur weekends par same $50/\$ impact hai. Lekin weekend shoppers zyada responsive ho sakte hain!

Interaction add karo:

interaction = ad_spend \times is_weekend

New model:

sales = 50 \cdot ad_spend + 200 \cdot is_weekend + 80 \cdot (ad_spend \times is_weekend) + 1000

Interpretation:

Weekday ( $is_weekend = 0$ ): sales $= 50 \cdot ad_spend + 1000$
- Har ad dollar → +$50 sales
Weekend ( $is_weekend = 1$ ): sales $= 50 \cdot ad_spend + 200 + 80 \cdot ad_spend + 1000$
- Simplify karo: sales $= 130 \cdot ad_spend + 1200$
- Har ad dollar → +$130 sales (2.6× zyada effective!)

Yeh step kyun? Interaction term $80 \cdot ad_spend \cdot is_weekend$ sirf weekends par activate hota hai, ad_spend effect mein extra slope add karta hai.

3. Ratio and Difference Features

Kab use karein:

Ratios: Scale se normalize karo (price-to-earnings ratio, clicks-per-impression)
Differences: Change capture karo (temperature delta, account balance change)
Rates: Efficiency metrics (revenue per employee, error per request)

Model seekh sakta hai: high debt → high risk. Lekin $80 k in co m e p a r$ 50k debt alag hai $30 k in co m e p a r$ 50k se!

Create karo:

debt_to_income_ratio = \frac{total_debt}{annual_income} = \frac{50000}{80000} = 0.625

Ab model relative burden dekhta hai, absolute numbers nahi. 0.625 ratio wala banda 1.8 ratio wale se safer hai, actual dollar amounts chahe kuch bhi hon.

Yeh step kyun? Division denominator se normalize karta hai, ek scale-invariant feature create karta hai.

4. Domain-Specific Features

Create karo:

price_lag1 = kal ka price
price_diff = aaj ka price - kal ka price (daily change)
rolling_mean_7 = pichhle 7 din ka average price (trend)
volatility = std(last 7 days) (risk measure)

Kyun? Yeh temporal patterns capture karte hain: trends, mean-reversion, volatility clustering.

Haversine distance create karo:

d = 2 r arcsin (sin^{2} (\frac{ϕ _{2} - ϕ _{1}}{2}) + cos (ϕ_{1}) cos (ϕ_{2}) sin^{2} (\frac{λ _{2} - λ _{1}}{2}))

jahan $ϕ$ = latitude, $λ$ = longitude, $r$ = Earth radius.

Kyun? Distance fare ka primary driver hai. Raw coordinates directly yeh encode nahi karte.

Common Mistakes

Common mistake Mistake 1: Creating Too Many Features (Combinatorial Explosion)

Galat idea: "Zyada features = zyada information, toh saari possible interactions create karo!" Yeh sahi kyun lagta hai: Polynomial degree 3, 4, 5... aur bhi complex patterns capture karega.

Sach yeh hai: $n = 10$ features ke saath degree 3 par, tumhe $(3 13) = 286$ features milte hain. Degree 4 → 1001 features. Tumne create kar liya hai:

Overfitting risk: Model polynomial combinations mein noise memorize karta hai
Computational cost: Training time $O (n^{2})$ ya $O (n^{3})$ kaafi algorithms ke liye
Multicollinearity: $x_{1}^{2}$ aur $x_{1} \cdot x_{2}$ highly correlated ho sakte hain Fix:

Domain knowledge use karo targeted interactions create karne ke liye
Regularization (L1/L2) apply karo useful features automatically select karne ke liye
Feature selection methods use karo (mutual information, recursive elimination)
Degree 2 se shuru karo, evaluate karo, phir cautiously degree 3 par jao agar zaroorat ho

Common mistake Mistake 2: Interaction Without Main Effects

Galat idea: $x_{1} \cdot x_{2}$ include karo lekin $x_{1}$ aur $x_{2}$ ko model se drop kar do.

Yeh sahi kyun lagta hai: Interaction term unka relationship capture karta hai, toh individual terms redundant hain.

Sach yeh hai: True model $y = 10 x_{1} + 5 x_{2} + 2 (x_{1} x_{2})$ consider karo. Agar tum sirf $y = β (x_{1} x_{2})$ fit karo:

Jab $x_{2} = 0$ : true effect $10 x_{1}$ hai, lekin tumhara model 0 predict karta hai (interaction vanish ho jaata hai!)
Tum model ko force kar rahe ho ki woh main effects ko interactions ke through explain kare, jo estimates ko distort karta hai

Fix: Hierarchical principle: Higher-order terms include karte waqt hamesha lower-order terms bhi include karo. Agar $x_{1} x_{2}$ model mein hai, toh $x_{1}$ aur $x_{2}$ bhi zaroor hone chahiye. Yeh model ko additive aur multiplicative effects alag karne ki flexibility deta hai.

Common mistake Mistake 3: Creating Features After Train-Test Split

Galat idea: Pehle data split karo, phir training set par category ke hisaab se mean(target) compute karo aur test set par apply karo.

Yeh sahi kyun lagta hai: Main sirf training data use karke mean compute kar raha hoon, toh koi leakage nahi.

Sach yeh hai: Leakage subtle ho sakta hai. Example: user_avg_purchase create karna us user ke historical purchases ke mean ke roop mein:

Agar tum split se pehle poore dataset par yeh compute karo, future test examples ke purchase amounts ne feature value ko influence kiya
Even agar tum sirf train par compute karo: agar wahi user train aur test dono mein appear karta hai, aur tum user-level behavior predict kar rahe ho, toh tumne test information leak kar di

Fix:

Pehle data split karo (ya temporal data ke liye time-based split use karo)
Aggregations/transformations sirf training data par compute karo
Test par wahi transformation apply karo training statistics use karke (jaise training mean, test mean nahi)
Time-series ke liye: future predict karne ke liye sirf past data use karo (lag features, rolling windows with proper cutoffs)

[!recall]- Ise Ek 12-Saal-Ke Bachche Ko Explain Karo

Socho tum predict karna chahte ho ki ek video game kitna fun hai, sirf do numbers use karke: graphics quality (1-10) aur story quality (1-10).

Agar tum bas inhe add kar do ( $fun = graphics + story$ ), tum keh rahe ho ki great graphics par no story wala game (10 + 2) utna hi fun hai jitna okay graphics aur story wala (6 + 6). Lekin yeh galat hai!

Feature creation aise ek NAYA number create karna hai jo capture kare jo tumhare liye actually matter karta hai:

$graphics \times story$ = ek "synergy score." Agar DONO achhe hain, toh yeh number huge hai (10 × 10 = 100). Agar ek terrible hai, chahe doosra great ho, yeh chhota hai (10 × 2 = 20). Yeh interaction capture karta hai ki dono ko milke achha hona chahiye.
$graphics^{2}$ capture karta hai ki graphics quality mein 9→10 jaana shayad 2→3 jaane se ZYADA matter kare (low end par diminishing returns, high end par excitement).

Purane numbers se yeh naye numbers create karke, tum computer ko woh patterns dekhne mein help karte ho jo woh pehle nahi dekh sakta tha. Yeh usse new "senses" dene jaisa hai duniya samajhne ke liye.

[!mnemonic] PRIME Features

Polynomial (curves ke liye powers) Ratios (scale se normalize karo) Interactions (joint effects, products) Mathematical (logs, sqrt, domain transforms) Engineered (lags, rolling, domain-specific)

Practical Implementation

import numpy as np
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import Ridge
 
# Sample data: predicting sales from ad_spend and website_traffic
X_train = np.array([[100, 500], [150, 800], [200, 1200], [80, 400]])
y_train = np.array([5000, 7500, 12000, 3800])
 
# Create polynomial features with interactions
poly = PolynomialFeatures(degree=2, include_bias=False, interaction_only=False)
X_poly = poly.fit_transform(X_train)
 
# Features are now: [ad, traffic, ad^2, ad*traffic, traffic^2]
print("Original shape:", X_train.shape)  # (4, 2)
print("Polynomial shape:", X_poly.shape)  # (4, 5)
print("Feature names:", poly.get_feature_names_out(['ad_spend', 'traffic']))
# ['ad_spend', 'traffic', 'ad_spend^2', 'ad_spend*traffic', 'traffic^2']
 
# Always scale after creating features!
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_poly)
 
# Train model
model = Ridge(alpha=1.0)
model.fit(X_scaled, y_train)
 
# Manual interaction creation example
X_manual = X_train.copy()
interaction = X_train[:, 0] * X_train[:, 1]  # ad_spend * traffic
X_with_interaction = np.column_stack([X_train, interaction])
print("With manual interaction:", X_with_interaction.shape)  # (4, 3)

When to Use Feature Creation

Polynomial features tab use karo jab:

Tumhare paas chhote feature sets hain (< 20 features) explosion se bachne ke liye
Nonlinear relationships suspect kiye jaate hain (physics, economics, biology)
Linear models use kar rahe ho jinhe curves capture karne mein help chahiye
Tumhare paas sufficient data hai ( $n ≫$ new feature count) overfitting se bachne ke liye

Interaction terms tab use karo jab:

Domain knowledge synergy suggest kare (medication + diet, temperature + humidity)
A/B testing different segments ke liye different effects reveal kare
Exploratory data analysis plots mein crossing effects dikhaye
Moderating variables model kar rahe ho (X ka Y par effect Z par depend karta hai)

Domain features tab use karo jab:

Tumhare paas temporal data hai (lags, rolling stats, seasonality)
Tumhare paas spatial data hai (distances, coordinates, neighborhoods)
Tumhare paas text/categorical data hai (encoding tricks, embeddings)
Known physical laws encode karne hain (conservation laws, rate equations)

Feature creation tab avoid karo jab:

Tree-based models (XGBoost, Random Forest) use kar rahe ho polynomial features ke liye—woh splits ke through interactions aur nonlinearity naturally capture karte hain
Tumhare paas bahut zyada features already hain (high-dimensional, genomics)—aur features cautiously add karo
Tumhara model already overfitting kar raha hai—pehle simplify karo, baad mein features add karo
Tumhare paas domain expertise nahi hai aur tum blindly features create kar rahe ho—yeh feature engineering roulette hai

Connections

Exploratory Data Analysis - feature ideas inspire karne ke liye relationships visualize karo
Feature Scaling and Normalization - polynomial/interaction features create karne ke baad hamesha scale karo
Regularization (L1, L2) - jab bahut saare features add kar liye hain toh overfitting control karo
Feature Selection Methods - irrelevant created features prune karo
Overfitting and Underfitting - zyada features overfitting risk badhate hain
Linear Regression - polynomial features linear models ko nonlinear bana dete hain
Decision Trees and Random Forests - interactions naturally capture karte hain, manual feature engineering ki kam zaroorat
Cross-Validation - evaluate karo ki naye features actually generalization improve karte hain ya nahi

#flashcards/ai-ml

Machine learning mein feature creation ka primary goal kya hai? :: Feature space ki expressive power badhana existing variables ko transform aur combine karke, taaki models woh relationships capture kar sakein jo raw data mein present nahi hain.

Ek linear model $y = w_{1} x_{1} + w_{2} x_{2} + b$ kabhi bhi multiplicative relationship $y = x_{1} \cdot x_{2}$ kyun nahi seekh sakta? :: Kyunki multiplication linear combinations ke hypothesis space mein nahi hai. Model sirf additive weighted sums seekh sakta hai, features ke products nahi. Interaction term $x_{3} = x_{1} \cdot x_{2}$ explicitly create karna yeh relationship add karta hai.

3 original features par degree-2 polynomial expansion apply karne se kitne features result hote hain?

(2 3 + 2) - 1 = (2 5) - 1 = 10 - 1 = 9

features (jisme 3 original linear terms, 3 squared terms, aur 3 pairwise interaction terms shamil hain).

Ek interaction term

x_{1} \times x_{2}

kya capture karta hai jo individual features

x_{1}

aur

x_{2}

nahi kar sakte?

Yeh capture karta hai ki ek feature ka effect doosre ki value par kaise depend karta hai (moderation/synergy).

x_{1}

ka coefficient effectively

(β_{1} + β_{3} x_{2})

ban jaata hai, jo

x_{2}

ke saath vary karta hai.

Absolute values ki jagah debt-to-income jaisi ratio features kyun create karni chahiye?

Ratios scale-invariant features create karte hain jo relative relationships capture karte hain.

30k vs \

80k income par $50k debt ka matlab alag hota hai—ratio denominator se normalize karke relative burden encode karta hai.

Interaction terms ke liye hierarchical principle bolo.

Higher-order terms include karte waqt hamesha lower-order terms bhi include karo. Agar

x_{1} x_{2}

tumhare model mein hai, toh

x_{1}

aur

x_{2}

dono bhi include hone chahiye taaki model additive aur multiplicative effects alag kar sake.

Polynomial ya interaction terms create karne ke baad features ko scale kyun karna zaroori hai?

Kyunki squared aur interaction features ki magnitudes bahut badi hoti hain (jaise

x_{1}^{2} \in [0, 10000]

agar

x_{1} \in [0, 100]

), jo ill-conditioned optimization landscapes create karta hai. Standardization saare features ko comparable scales par laata hai stable gradient descent ke liye.

Tree-based models jaise Random Forest ke liye polynomial features create karna kyun avoid karna chahiye?

Tree-based models single features par recursive splits ke through nonlinearity aur interactions naturally capture karte hain. Polynomial features add karna redundant hai aur dimensionality badhata hai bina kisi fayde ke, potentially training slow karta hai.

Sahi order kya hai: pehle data split karo phir features create karo, ya pehle features create karo phir split karo?

Pehle data SPLIT karo, phir sirf training data statistics use karke features create karo. Split se pehle poore dataset par aggregations compute karna target leakage cause karta hai—test information feature values ko influence karti hai.

Time-series prediction ke liye price_lag_1 aur rolling averages jaisi lag features kyun create karte hain?

Yeh temporal dependencies capture karte hain: lag features autocorrelation encode karte hain (aaj kal par depend karta hai), rolling averages trends capture karte hain, aur rolling standard deviations volatility capture karte hain. Raw current values mein yeh temporal context nahi hota.