Learn about overfitting and curve fitting

6.1.10 · Stock-Market › Algorithmic & Quant Trading

Overfitting Kya Hai?

Yeh KYUN hoti hai:

1. Bahut zyada parameters: 50 tunable parameters wala model kisi bhi dataset ko fit kar sakta hai, chahe random noise ho
2. Insufficient data: 100 data points se 20 parameters reliably estimate nahi ho sakte (rule of thumb: har parameter ke liye 10-20 observations chahiye)
3. Repeated testing: 100 backtests chalana aur best ek chunna matlab tumne best strategy nahi, balki sabse lucky parameter set dhundha
4. Regime changes ko ignore karna: 2010-2020 pe trained model 2021 mein fail ho sakta hai agar market structure change ho gayi

Overfitting ki Mathematics

Bias-Variance Tradeoff

Har model ka prediction error teen components mein decompose hota hai:

$\text{Total Error}={\text{Bias}}^2+\text{Variance}+\text{Irreducible Error}$

First principles se derive karte hain:

Maan lo ki true relationship hai $y=f(x)+ϵ$ jahan $ϵ$ random noise hai jiske liye $E[ϵ]=0$ aur $\text{Var}(ϵ)={\sigma }^2$.

Hamara model predict karta hai $\hat{f}(x)$. Point $x_0$ pe expected prediction error hai:

$E[(y-\hat{f}(x_0){)}^2]$

Step 1: $y=f(x_0)+ϵ$ use karke expand karo:

$E[(f(x_0)+ϵ-\hat{f}(x_0){)}^2]$

Step 2: Rearrange karo aur $E[\hat{f}(x_0)]$ add/subtract karo:

$E[((f(x_0)-E[\hat{f}(x_0)])+(E[\hat{f}(x_0)]-\hat{f}(x_0))+ϵ{)}^2]$

Step 3: Square expand karo (teen terms, chhe cross products):

$E[(f(x_0)-E[\hat{f}(x_0)]{)}^2]+E[(E[\hat{f}(x_0)]-\hat{f}(x_0){)}^2]+E[{ϵ}^2]+\text{cross terms}$

Cross terms KYUN vanish hote hain:

• $(f(x_0)-E[\hat{f}(x_0)])$ ek constant hai (koi randomness nahi)
• $E[ϵ]=0$ assumption ke by
• $E[\hat{f}(x_0)]-\hat{f}(x_0)$ ka mean expectation ki definition ke by zero hota hai

Step 4: Components ko pehchano:

$\underset{{\text{Bias}}^2}{\underbrace{(f(x_0)-E[\hat{f}(x_0)]{)}^2}}+\underset{\text{Variance}}{\underbrace{E[(\hat{f}(x_0)-E[\hat{f}(x_0)]{)}^2]}}+\underset{\text{Irreducible}}{\underbrace{{\sigma }^2}}$

Overfitting in Action: Polynomial Regression

Maan lo hum degree $d$ ke polynomial se stock returns predict kar rahe hain:

$\hat{y}={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2+\dots +{\beta }_d{x}^d$

Yeh overfitting KYUN demonstrate karta hai: Jaise jaise $d$ badhta hai, model training data ko perfectly fit kar sakta hai, lekin test error explode ho jaata hai.

Mathematically HOW dekhen:

$n$ data points aur degree $d=n-1$ ke saath, polynomial har point se exactly pass hota hai (Lagrange interpolation). Training error = 0, lekin points ke beech mein polynomial wildly oscillate karta hai.

Degrees of freedom: Effective number of parameters. Polynomial degree $d$ ke liye, DoF = $d+1$. Rule of thumb: stable estimates ke liye $n\ge 10\times \text{DoF}$ chahiye.

Algorithmic Trading ke Examples

Common Mistakes aur Unhe Kaise Avoid Karein

Overfitting Detect aur Prevent Kaise Karein

Detection Methods

1. Train/Test Split: $\text{Overfitting Score}=\frac{\text{Train Performance}-\text{Test Performance}}{\text{Train Performance}}$

   Agar yeh ratio > 0.2 (20% degradation) ho, toh likely overfit hai.

2. Cross-Validation:

   • K-Fold: Data ko K chunks mein split karo, K-1 pe train karo, 1 pe test karo, K baar repeat karo
   • Time-series CV: Expanding ya sliding window use karo (temporal order preserve karo) Yeh KYUN kaam karta hai: K alag test sets pe simultaneously overfit karna mushkil hai. Folds mein high variance = unstable model.

3. Learning Curves:

   • Training error aur validation error ko training set size ke against plot karo
   • Converging curves (large N pe chhota gap): Achha generalization
   • Diverging curves (widening gap): Overfitting

   HOW interpret karein: Agar zyada data add karne se gap close hota hai, toh zyada data collect karo. Agar gap persist kare, toh model simplify karo.

4. Permutation Test:

   • Target variable ko randomly shuffle karo (koi bhi real relationship destroy karo)
   • Poora optimization process dobara chalao
   • Agar shuffled data pe "optimized" strategy abhi bhi achhi lagti hai, tumhara process noise ko curve-fit kar raha hai

   Yeh KYUN powerful hai: Random data mein koi signal nahi hota. Koi bhi found edge = tumhari methodology se false positive.

Prevention Strategies

Practical Prevention Checklist:

1. Pehle simplicity: Simple model se shuru karo (linear regression, 2-3 features). Complexity sirf tabhi add karo jab theory aur out-of-sample improvement se justified ho.

2. Feature engineering over feature dumping: 5 carefully chosen features 100 raw indicators se behtar hain.

3. Economic reasoning: Har parameter ka financial explanation hona chahiye. "23-day MA cross kare 47-day MA ko jab buy karo" ka koi theoretical basis nahi—kyun 22 aur 46 nahi?

4. Walk-forward analysis:

   • Year 1 pe train karo, Year 2 pe test karo
   • Year 1-2 pe retrain karo, Year 3 pe test karo
   • Rolling forward continue karo
   • KYUN: Real trading simulate karta hai jahan tum periodically naye data ke saath retrain karte ho

5. Robust optimization: Single best parameter set chunne ki jagah, woh chunno jo nearby parameters ki ek range mein kaafi achha perform kare. Agar performance chhote changes ke liye sensitive hai (jaise MA period 14 vs. 15 completely results change kar de), strategy fragile hai.

6. Information coefficient (IC): Predictive models ke liye, predictions aur actual outcomes ke beech rank correlation measure karo: $\text{IC}=\text{cor}(\text{predicted returns},\text{actual returns})$

   IC time periods mein stable rehna chahiye. Declining IC = strategy decay ya overfitting.

7. Sample size rules:

   • Minimum: $n>10p$ jahan $p$ = number of parameters
   • Conservative: $n>30p$
   • Rare events ke liye (jaise crashes predict karna): Hundreds of occurrences chahiye

Mathematical Rigor: Degrees of Freedom

Ek model ke effective degrees of freedom woh independent parameters ki sankhya hai jo estimate ki ja rahi hain. Trading strategies ke liye:

$\text{DoF}=\text{\# explicit parameters}+\text{\# implicit choices}$

Explicit: MA periods, threshold values, position sizes Implicit: Kaun se features include karein, kaun sa optimization metric use karein, backtest kab start/stop karein

Yeh KYUN matter karta hai: Statistical tests degrees of freedom assume karte hain. Agar tum claim karo "Meri strategy ka t-statistic 3.5 hai, p < 0.001 pe significant hai," lekin tumne secretly 100 variations try kiye, toh actual p-value bahut zyada hai.

Bonferroni correction: Agar tum $m$ independent tests chalate ho, significance level adjust karo:

${\alpha }_{\text{corrected}}=\frac{\alpha }{m}$

Agar tum overall 5% significance chahte ho aur 100 tests chalate ho: ${\alpha }_{\text{corrected}}=0.05/100=0.005$. Ab tumhe 1.96 ki jagah ~3.5 ka t-statistic chahiye.

Connections

• 6.1.8-Backtesting-strategies-and-interpreting-results - Overfitting backtest results ko unreliable banata hai
• 6.1.9-Walk-forward-analysis-and-out-of-sample-testing - Overfitting ke khilaf primary defense
• 6.21-What-is-quantitative-trading - Quant models especially overfitting ke liye vulnerable hain
• 5.3.5-Risk-management-in-algorithmic-trading - Overfit models ka real-world risk backtest se zyada hota hai
• 6.1.3-Types-of-algorithmic-strategies - Har strategy type ko overfitting se bachna chahiye
• 4.4-Standard-deviation-and-variance - Variance ko samajhna bias-variance tradeoff ki key hai
• Machine-learning-in-trading - ML models ki high capacity hoti hai, isliye strong overfitting prevention zaroori hai

#flashcards/stock-market

Overfitting trading mein kya hoti hai? :: Jab model historical data mein noise aur random fluctuations seekh leta hai na ki true signal, jisse backtest performance achhi hoti hai lekin live performance kharab.

Overfitting HOW detect karte hain? :: Train vs test performance compare karo (>20% degradation warning hai), k-fold cross-validation use karo (folds mein high variance = unstable model), learning curves plot karo (diverging curves), shuffled data pe permutation tests chalao.

Zyada data overfitting KYUN hamesha nahi rokta? :: Markets mein regime changes aur non-stationarity hoti hai. Alag regimes ke purane data se zyada nuksan ho sakta hai. Current regime ka 1 saal data multiple alag regimes spanning 10 saal se behtar hai.

Concept Map