Understand in-sample vs out-of-sample testing

6.2.7 · Stock-Market › Backtesting Frameworks

In-Sample aur Out-of-Sample kya hote hain?

Yeh split KYUN? Kyunki strategies trial-and-error se banti hain. Har tweak jo tum in-sample performance improve karne ke liye karte ho, woh random noise fit karne ka risk leta hai. Out-of-sample tumhara reality check hai: kya edge tab bhi bana rehta hai jab strategy naye data ka samna karti hai?

Iske BINA kya hota hai? Tumhe curve-fitting milti hai: ek strategy jiske 15 parameters hain, in-sample 80% win-rate hai, lekin jis moment tum ise live trade karte ho woh paisa duboti hai. Classic quant graveyard yahi hai.

Split KAISE karein? Common approaches:

Chronological split: Pehle 70% = IS, aakhri 30% = OOS (time-series nature ka respect karta hai)
Walk-forward: Rolling IS/OOS windows (zyada robust, advanced backtesting mein cover hota hai)
Leave-out segments: Specific market regimes (jaise 2008 crisis) ko OOS ke roop mein hold out karo

Math: Overfitting Kyun Hoti Hai

Chalte hain derive karte hain ki zyada parameters → overfitting ka zyada risk kyun hota hai.

Setup: Tumhare paas $N$ data points hain (jaise daily returns). Tum $p$ parameters wala model fit karte ho.

Degrees of Freedom: Model data fit karne ke liye $p$ degrees of freedom "use up" karta hai. Genuine signal ke liye remaining degrees of freedom: $DoF = N - p$ .

Signal vs Noise: Maano returns = signal + noise, $r_{t} = s_{t} + ϵ_{t}$ , jahan $ϵ_{t} \sim N (0, σ^{2})$ random hai.

Jab tum $N$ points par $p$ parameters optimize karte ho:

Expected in-sample error: Tumhara optimizer $s_{t} + ϵ_{t}$ ka best fit dhundhta hai, jisme kuch noise fit karna bhi shamil hai.
Expected out-of-sample error: OS data mein alag noise $ϵ_{t}^{'}$ hoti hai, isliye noise-fitted parts fail ho jaate hain.

Example numbers:

IS period: 252 days (1 saal), $N = 252$
Strategy ke 10 parameters hain (moving average lengths, thresholds, stop-losses, etc.), $p = 10$
Ratio: $p / N = 10/252 \approx 4%$ . Borderline acceptable.
Agar $p = 50$ , ratio = 20% → severe overfitting risk. Tum noise fit kar rahe ho.

Worked Example: Moving Average Crossover

Strategy: Buy karo jab 20-day MA, 50-day MA ke upar cross kare. Sell karo jab neeche cross kare.

Step 1: In-Sample Optimization

Tumhare paas SPY ka 5 saal ka data hai (2017-2021, ~1260 days). Tum sabhi combinations test karte ho:

Short MA: 10, 15, 20, 25, 30 days
Long MA: 40, 50, 60, 70, 80 days

Yeh $5 \times 5 = 25$ combinations hain (25 parameters tested).

Result: Best IS combo hai (15, 70) jiske saath 18% annual return.

Yeh step kyun? Tum woh parameter set dhundh rahe ho jo IS performance maximize kare. Yeh standard optimization hai.

Step 2: Out-of-Sample Test

2022-2023 (~500 days) ko OOS ke roop mein reserve karo. Tum ne optimization ke dauran yeh data kabhi nahi chhua.

(15, 70) strategy ko OS data par run karo.

Result: OS return hai 3% annualized.

Drop kyun? (15, 70) combo ne probably 2017-2021 ke random SPY fluctuations exploit kiye jo 2022-2023 mein repeat nahi hue. Classic overfitting.

Interpret KAISE karein?

Metric	In-Sample	Out-of-Sample	Gap
Annual Return	18%	3%	15%
Sharpe Ratio	1.2	0.3	0.9

Bada gap = overfitting. Strategy ka koi real edge nahi hai; usne IS noise memorize kar li.

Agar OS, IS jaise hota toh? Maano OS return = 14%. Gap = 4%. Phir bhi kuch overfitting hai, lekin edge real hai. 20-30% degradation typical hai (parameters kabhi perfectly tuned nahi hote). 80% collapse (18% → 3%) overfitting chillata hai.

Worked Example 2: Mean Reversion with Multiple Filters

Strategy: Buy karo jab RSI < 30 AND price < 200-day MA AND volume > 1.5× average.

Parameters:

RSI threshold: $θ_{1} \in {20, 25, 30, 35}$
MA length: $θ_{2} \in {150, 200, 250}$
Volume multiplier: $θ_{3} \in {1.2, 1.5, 2.0}$

Total combinations: $4 \times 3 = 36$ parameter sets.

Data: 10 saal AAPL (2013-2022), 70/30 split → IS: 2013-2019 (7 saal), OOS: 2020-2022 (3 saal).

Step 1: In-Sample Grid Search

Sabhi 36 combos test karo. Best IS: RSI=25, MA=150, Vol=2.0 → 22% annual return, Sharpe 1.5.

Yeh step kyun? Tum hyperparameter optimization (aka parameter tuning) kar rahe ho. Har backtest platform yahi karta hai.

Step 2: Out-of-Sample Reality Check

(25, 150, 2.0) ko OOS 2020-2022 par run karo.

Result: 8% return, Sharpe 0.6.

Drop kyun?

2020 mein COVID crash aaya (regime change)
(25, 150, 2.0) combo 2013-2019 ki "normal" volatility ke liye tune hua tha
Pandemic trading mein volume spike behavior badal gaya

Fix KAISE karein?

Walk-forward testing (har saal rolling IS window use karke re-optimize karo)
Regularization: Complex strategies ko penalize karo (kam parameters)
Ensemble: "Best" ek pick karne ki jagah multiple parameter sets ka average lo

Key insight: 70/30 split ke baad bhi, agar OS mein regime shift ho (2020 pandemic), tumhari strategy struggle karegi. OS representative bhi hona chahiye aur unseen bhi.

Common Mistakes (Steel-man)

Common mistake Mistake 2: "Maine sirf ek baar OS data dekha tweak karne ke liye, phir re-split kiya"

Kyun sahi lagta hai: Ek peek se kuch nahi hoga, aur phir tum ek "naya" OOS set bana lete ho.

Problem: Jis moment tum parameters adjust karne ke liye OOS feedback use karte ho, OS in-sample ban jaata hai. Tumhara dimaag ab us data ke liye overfit ho gaya hai. Re-splitting se yeh mita nahi—tumne data contaminate kar diya.

Steel-man: Academic ML mein, tumhare paas IS/validation/test splits ho sakte hain, jahan validation tuning ke liye hota hai. Lekin backtesting mein, zyaatar traders ke paas sirf IS/OS hota hai, aur OOS pristine rehna chahiye.

Fix: Development 100% complete hone tak OOS kabhi mat dekho. Agar iterate karna hi pade, sirf IS ke andar walk-forward ya cross-validation use karo.

Practical Implementation

Python (pandas) mein split KAISE karein:

import pandas as pd
 
df = pd.read_csv('AAPL_daily.csv', parse_dates=['Date'])
df = df.sort_values('Date')
 
split_date = '2020-01-01'
is_data = df['Date'] < split_date]  # In-sample
oos_data = df[df['Date'] >= split_date]  # Out-of-sample
 
print(f"IS: {len(is_data)} days, OOS: {len(oos_data)} days")

Chronological KYUN? Time-series data mein temporal dependencies hoti hain. Shuffling (jaise ML mein) ise todta hai—tum "future" data par train karke "past" predict kar rahe hoge. Chronological split causality ka respect karta hai.

Walk-forward ke baare mein KYA?

# Pseudo-code for walk-forward
results = []
for start in range(0, len(df), 252):  # Roll every year
    is_window = df[start : start+756]  # 3 years IS
    oos_window = df[start+756 : start+1008]  # 1 year OOS
    
    best_params = optimize(strategy, is_window)
    oos_perf = backtest(strategy, oos_window, best_params)
    results.append(oos_perf)
 
avg_oos = np.mean(results)  # Aggregate OOS performance

Better KYUN? Multiple OOS periods luck ko reduce karte hain. Agar strategy 5 OS windows mein kaam kare, toh yeh ek lucky OS period se zyada robust hai.

80/20 Core

20% concepts jo IS/OOS testing ka 80% explain karte hain:

IS = jahan tum optimize karte ho, OOS = jahan tum validate karte ho. Inhe kabhi mix mat karo.
Overfitting $p / N$ ratio ke saath badhti hai (parameters / data points). $p / N < 5%$ rakho.
Bada IS/OOS performance gap (>40% drop) = overfitting. Strategy discard karo.
Walk-forward > single split robust validation ke liye (multiple OOS tests).

Agar yeh chaar yaad rahe, tum 80% backtesting disasters se bach jaoge.

Diagram

Diagram kya dikhata hai:

Top panel: IS (blue) vs OOS (red) mein equity curve. Dekho IS smooth hai (overfit), OOS noisy hai (reality).
Bottom panel: Parameter sensitivity. IS mein optimal param (green dot) OOS mein poor performance deta hai (red dot). Overfit.

Recall Feynman Technique: Ek 12-saal ke bacche ko samjhao

Socho tum exam ki taiyaari kar rahe ho. Tumhe 100 practice problems milte hain (in-sample). Tum har trick try karte ho, patterns memorize karte ho, aur 95% score karte ho. Lekin phir real test (out-of-sample) mein alag problems hain—milte-julte, lekin same nahi. Tum sirf 60% score karte ho. Kyun? Tumne practice problems memorize kar li, underlying math nahi seekhi. Trading strategies bhi aise hi hain. In-sample tumhara practice set hai jahan tum strategy tune karte ho. Out-of-sample naye data ke saath real test hai. Agar tumhari strategy in-sample mein 95% score kare lekin out-of-sample mein 60%, usne random noise memorize kar li (jaise practice problems word-for-word yaad kar lena). Ek achhi strategy real patterns seekhti hai aur dono mein 80% score karti hai—perfect nahi, lekin consistent.

Rule: Out-of-sample "test answers" practice karte waqt kabhi mat dekho. Ek baar dekh liya toh un-see nahi kar sakte, aur tumhari practice contaminated ho jaati hai.