Understand backtesting strategies properly

4.8.8 · Stock-Market › Trading Psychology

Overview

Backtesting ek aisa process hai jisme trading strategy ko historical data par test kiya jaata hai — yeh dekhne ke liye ki woh past mein kaisi perform karti. Yeh ek trading idea aur real capital deploy karne ke beech ka bridge hai, lekin yahi jagah hai jahan zyaadatar traders khud ko jhoothe confidence mein fool kar lete hain.

What Backtesting Actually Tests

WHY We Backtest

1. Validate Logic: Kya strategy ki core hypothesis (jaise, "momentum 20 days tak persist karta hai") actually historical data mein dikhti hai?
2. Quantify Performance: Returns, drawdowns, win rates ke liye realistic expectations kya hain?
3. Build Confidence: Live trading mein strategy rough patch mein aane par emotional doubt kam hota hai
4. Parameter Optimization: Reasonable bounds ke andar best settings (jaise, moving average periods) dhundhna

WHAT Backtesting Cannot Do

• Predict the future: Markets evolve karte hain. Jo strategy 2010-2020 mein kaam aayi, woh 2021-2026 mein fail ho sakti hai
• Account for execution reality: Slippage, liquidity, aur tumhari khud ki emotions backtest mein nahi hoti
• Guarantee profitability: Past performance ≠ future results (yeh legally required disclosure hai — ek wajah se)

The Mathematics of Backtest Reliability

Sample Size and Statistical Significance

Yeh determine karne ke liye ki strategy ka edge real hai ya luck, humein statistical significance chahiye. Required sample size tumhare desired confidence level par depend karta hai.

Derivation from scratch:

Maano har trade independent hai with win probability $p$ (unknown). Tumne $n$ trades observe kiye jisme $w$ wins hain. Sample win rate hai $\hat{p}=w/n$.

Is proportion ka standard error hai: $SE=\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$

95% confidence interval ke liye (1.96 standard deviations): $\text{CI}=\hat{p}\pm 1.96\cdot SE$

Agar tumhari strategy 55% win rate claim karti hai, tum chahte ho ki confidence interval 50% (break-even) ko include na kare. Inequality set up karo:

$\hat{p}-1.96\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}>0.50$

$\hat{p}=0.55$ ke liye $n$ solve karo: $0.55-1.96\sqrt{\frac{0.55\cdot 0.45}{n}}>0.50$ $0.05>1.96\sqrt{\frac{0.2475}{n}}$ $0.05^2>(1.96{)}^2\cdot \frac{0.2475}{n}$ $n>\frac{3.8416\cdot 0.2475}{0.0025}\approx 380\text{ trades}$

WHY this matters: Sirf 50 trades ke saath, tumhara 55% win rate easily noise ho sakta hai. Statistical confidence claim karne ke liye hundreds of trades chahiye.

Tum Nifty 50 stocks par ek swing trading strategy backtest karte ho (2015-2024):

• Total trades: 85
• Wins: 50 (58.8% win rate)
• Avg profit per trade: ₹2,100

Question: Kya yeh strategy proven hai?

Analysis: $SE=\sqrt{\frac{0.588\cdot 0.412}{85}}=\sqrt{\frac{0.242}{85}}=0.053$ $\text{95\% CI}=0.588\pm 1.96(0.053)=[0.484,0.692]$

Confidence interval 50% ko include karta hai, matlab tum confidently nahi keh sakte ki is strategy ka koi edge hai. Tum shayad sirf random variation dekh rahe ho.

WHY this step? Hum check kar rahe hain ki observed win rate coin-flip luck se alag hai ya nahi.

Example 2: Sufficient Sample

Wahi strategy, lekin 200 stocks mein 10 saal ka daily data:

• Total trades: 450
• Wins: 265 (58.9% win rate)

$SE=\sqrt{\frac{0.589\cdot 0.411}{450}}=0.023$ $\text{95\% CI}=0.589\pm 1.96(0.023)=[0.544,0.634]$

Ab interval 50% ko exclude karta hai — yeh edge statistically significant hai.

Look-Ahead Bias: The Silent Killer

Common forms:

1. Using adjusted prices incorrectly: Stock splits ko historical data mein backward adjust kiya jaata hai. Agar tum adjusted data par "buy when price < ₹100" jaise rule backtest karte ho, tum future split information use kar rahe ho
2. Survivorship bias: Sirf woh stocks test karna jo aaj bhi exist karte hain (survivors) — bankrupt/delisted stocks ko ignore karna
3. Peeking at the close: "Jab price MA cross kare" signal generate karna lekin decide karne ke liye usi din ka closing price use karna — reality mein, tum close market close hone ke baad hi jaante ho

Flawed backtest: "Reliance ko buy karo jab adjusted close < ₹200 ho, ₹250 par sell karo"

Tum aise data par yeh run karte ho jahan Reliance ka 2017 mein 1:1 split hua tha. Backtest dikhata hai ki tumne 2015 mein (adjusted) ₹180 par "buy" kiya aur 2016 mein (adjusted) ₹250 par sell kiya. Bahut bada profit!

WHY this is wrong: 2015 mein actual price ₹360 thi (split adjustment se pehle). Tum kabhi nahi khareedte kyunki tumhara rule kehta tha "< ₹200" — us waqt ki real price ke hisaab se, backward-adjusted price ke hisaab se nahi.

The fix: Point-in-time pricing use karo, ya pre/post-split analysis clearly alag karo.

Overfitting: Torturing the Data Until It Confesses

The Mathematics of Overfitting

Agar tum $k$ independent strategies test karte ho, jisme har ek ko luck se "significant" result dikhane ki 5% chance hai (p < 0.05), toh probability ki kam se kam ek strategy chance se acchi lagegi:

$P(\text{at least one false positive})=1-(0.95{)}^k$

Derivation:

• Probability ki ek test false positive na de: 0.95
• Probability ki saare $k$ tests false positive na dein: $(0.95{)}^k$
• Probability ki kam se kam ek false positive ho: upar ka complement

$k=20$ tests ke liye: $1-(0.95{)}^{20}=0.64$ (64% chance of fool's gold!)

Tum ek moving average crossover strategy test karte ho:

• Fast MA: 5, 10, 15, 20, 25, 30 days (6 choices)
• Slow MA: 50, 100, 150, 200 days (4 choices)
• Stop loss: 2%, 3%, 5%, 7%, 10% (5 choices)

Total combinations: $6\times 4\times 5=120$

Tumhe milta hai ki (15, 100, 3%) 2015-2020 mein 32% annual return aur max drawdown sirf 8% deta hai. Amazing!

WHY this is dangerous: Tumne 120 strategies test kiin. False discovery formula se: $P(\text{at least one looks good by chance})=1-(0.95{)}^{120}\approx 0.998$

Tumhe almost guarantee hai ki kuch aisa milega jo sirf luck se accha lagta hai. Yahi curve-fitting ya data mining hai.

WHY this step? Hum calculate kar rahe hain ki probability kya hai ki hamara "best" result itni saari variations test karne ki wajah se ek illusion hai.

Out-of-Sample Testing: The Honest Judge

WHAT this does: Agar performance out-of-sample mein gir jaata hai, toh tumhari strategy overfit thi.

HOW to implement:

In-sample: Develop strategy, optimize parameters
[Freeze the strategy completely]
Out-of-sample: Run once, record results, no tweaking

Agar tum out-of-sample results dekhne ke baad in-sample mein wapas jaate ho, tumne test contaminate kar diya.

Single train/test split se zyaada robust:

Method:

1. 2010-2012 par train karo, 2013 par test karo (results record karo)
2. 2011-2013 par train karo, 2014 par test karo (results record karo)
3. 2012-2014 par train karo, 2015 par test karo (results record karo)
4. Aage rolling forward jaari rakho...

Saare out-of-sample test results aggregate karo. Agar strategy time ke saath degrade hoti hai, toh woh market regime changes ke saath adapt nahi ho rahi.

WHY this step? Markets evolve karte hain. Jo strategy ek period mein kaam karti hai, woh doosre mein fail ho sakti hai. Walk-forward tumhe yeh degradation real-time simulation mein dikhata hai.

Transaction Costs: The Returns Killer

$R_{\text{net}}=R_{\text{gross}}-\text{Costs}$

Jahan: $\text{Costs}=n\times (\text{Commission}+\text{Slippage}+\text{Impact})$

• $n$ = number of trades
• Commission = brokerage fees (₹20 per trade, ya 0.03% of value)
• Slippage = expected price aur actual execution price ke beech ka difference
• Impact = tumhare khud ke order size ki wajah se price tumhare against move hona (large orders ke liye relevant)

Derivation from first principles:

Tumhara backtest 120 trades/year mein 25% gross annual return dikhata hai. Tumhare paas ₹10,00,000 capital hai.

Maano:

• Average trade size: ₹50,000
• Commission: 0.03% each side (buy + sell) = 0.06% per round trip
• Slippage: 0.1% per side = 0.2% per round trip
• Total cost per trade: trade value ka 0.26%

$\text{Cost per trade}=0.0026\times 50,000=₹130$ $\text{Annual costs}=120\times 130=₹15,600$

Tumhara backtest ₹2,50,000 profit dikhata hai (₹10,00,000 ka 25%).

$R_{\text{net}}=\frac{250,000-15,600}{1,000,000}=23.44\%$

WHY this matters: High-frequency strategies jisme profit per trade kam hota hai, costs ki wajah se barbaad ho jaati hain. Ek strategy jisme 200 bps gross profit per trade aur 50 bps costs hain, apna 25% edge gawa deti hai.

Scalping strategy backtest:

• Gross profit per trade: ₹500
• Trades per day: 10
• Trading days per year: 250
• Commission + slippage per trade: ₹80

Gross annual: $500\times 10\times 250=₹12,50,000$ Costs: $80\times 10\times 250=₹2,00,000$ Net annual: $₹10,50,000$

Costs ne gross profit ka 16% consume kar liya. Agar tumhara backtest realistic costs include nahi karta tha, tumne returns 16% overestimate kar liye.

WHY this step? Hum dikh raha kar rahe hain ki transaction costs koi chhota adjustment nahi hain — yeh fundamentally strategy viability ko change karte hain, especially high-turnover approaches ke liye.

Psychological Realism: You Are Not a Robot

Why it feels right: Screen par numbers emotionally neutral hote hain. 18% sirf ek statistic hai.

The Reality: Jab tum real money mein 18% down hote ho, dekh rahe ho ki tumhara ₹5,00,000 ₹4,10,000 ban gaya, tumne sirf statistics nahi khoye — tumne ek nayi car, ek vacation, apna confidence khoya. Rules override karne ki urge overwhelming ho jaati hai.

Steel-man: Tum aisa feel karne ke liye stupid nahi ho. Loss aversion evolutionary hai — hamare ancestors jo resource loss par panic karte the, woh survive karte the. Market is instinct ko punish karti hai.

The Fix:

1. Backtest se chhoti position size trade karo. Agar backtest 10% per position kehta hai, 5% trade karo
2. Live jaane se pehle 3 mahine tak strategy ko paper trade karo. Bina paise ke real-time mein emotional drawdown experience karo
3. Apne rules aur backtested max drawdown likhke rakho. Jab 15% hit karo, unhe padho. Poori tarah follow karne ya completely band karne ka commitment karo — beech mein koi tweaking nahi

Key Metrics to Evaluate

$\text{Sharpe}=\frac{R_p-R_f}{{\sigma }_p}$

• $R_p$ = portfolio return
• $R_f$ = risk-free rate (jaise, Indian government bonds ke liye 7%)
• ${\sigma }_p$ = portfolio returns ka standard deviation

WHY Sharpe matters: 30% return aur 40% volatility wali strategy (Sharpe = 0.58) 20% return aur 10% volatility wali strategy (Sharpe = 1.3) se buri hai. Tum risk per unit par returns chahte ho.

Maximum Drawdown (MDD):

$\text{MDD}={\max }_t\left(\frac{{\text{Peak}}_t-{\text{Trough}}_t}{{\text{Peak}}_t}\right)$

Yeh saare time points $t$ mein measured sabse bada peak-to-valley loss hai. Agar tumhara peak ₹10,00,000 tha aur worst subsequent trough ₹7,00,000 tha, toh $\text{MDD}=(10,00,000-7,00,000)/10,00,000=30\%$.

WHY MDD matters: Yahi woh dard hai jo tum feel karoge. Agar tum 30% loss stomach nahi kar sakte, yeh strategy mat trade karo.

Profit Factor:

$\text{PF}=\frac{\sum \text{Winning Trades}}{\sum |\text{Losing Trades}|}$

PF 1.5 matlab har ₹1 lose karne par tum ₹1.50 banate ho. 1.0 se neeche matlab tum paise kho rahe ho.

Strategy results (2015-2024):

• CAGR: 22%
• Max drawdown: 25%
• Sharpe ratio: 1.1
• Profit factor: 1.8
• Win rate: 52%
• Avg win: ₹4,500
• Avg loss: ₹3,200
• Total trades: 380

Analysis:

• Sample size: 380 trades ✓ (pehle ke formula ke hisaab se sufficient)
• Sharpe 1.1: Decent risk-adjusted return
• MDD 25%: Kya tum ₹10,00,000 ke account par ₹2,50,000 down hona handle kar sakte ho? Honest raho.
• PF 1.8: Solid — lagbhag 2:1 gross profit to loss ratio
• Win rate 52%: Thoda sa edge, reasonable

Verdict: Statistically valid, lekin tumhari personal risk tolerance hi final judge hai.

Common Backtesting Mistakes

Why it feels right: Historical data providers tumhe adjusted prices dete hain, convenient!

The trap: Tum future knowledge (splits, dividends) use kar rahe ho jo signal time par exist nahi karti thi.

Fix: Signals ke liye raw prices use karo, ya ensure karo ki tumhara adjustment strictly point-in-time ho.

Mistake 2: Ignoring Liquidity

Why it feels right: Tumhara backtest dikhata hai tumne ₹12 par ek penny stock ke 10,000 shares kharide.

The trap: Reality mein, us din ₹12 par sirf 500 shares available the. Tumhara order fill karte karte price ₹14 ho jaati.

Fix: Minimum average daily volume se neeche ke stocks filter karo (jaise, ₹10 lakh ADV). Slippage conservatively model karo.

Mistake 3: Peeking Into the Future

Why it feels right: "Main buy karunga jab price yesterday's high ke upar cross kare, aaj ke close use karke."

The trap: Tum aaj ka close 3:30pm tak jaante nahi. Agar cross 10am par hua, tumne kaun sa price use kiya?

Fix: Open-to-close logic carefully use karo. Intraday signals mein intraday data use karo. EOD signals mein previous day ka data use karo.

"Yaad hai jab tum apne purane Fortnite matches replay karte the dekhne ke liye ki tumne kya galat kiya? Backtesting trading ke liye bilkul waisa hi hai. Tum purane market data par pretend trade karte ho yeh dekhne ke liye ki tumhari strategy kaam karti ya nahi.

Lekin yahan trick hai: ek match jeetna matlab yeh nahi ki same strategy se agli baar bhi jeetoge. Shayad doosre players kacche the, ya tumhe lucky loot mili.

Backtesting bhi waisa hi hai. Tumhe ensure karna hai:

1. Tumne cheat nahi kiya future jaanke (jaise replay karna but pehle se jaanna ki enemies kahan spawn honge)
2. Tumne itne matches khele ki pata chale yeh luck nahi thi (ek achha game tumhe pro nahi banata)
3. Tum lag aur bugs account karo (trading mein slippage aur costs)
4. Tum honest raho ki pressure mein actually woh moves karoge ya nahi (emotions real hoti hain)

Agar tumhara backtest kehta hai ki tum ₹10 lakh banate, lekin tumne 100 alag strategies test kiin aur best wali pick ki, tum shayad sirf sabse lucky game dhundhe. Woh luck repeat nahi hogi."

Connections

• 4.8.01-Confirmation-bias-in-trading: Backtesting confirmation bias ko feed kar sakta hai agar tum timeframes cherry-pick karo
• 4.8.05-Overconfidence-from-past-wins: Ek achha backtest dangerous overconfidence create karta hai
• 4.5.03-Position-sizing-and-risk-per-trade: Backtest expected drawdown batata hai, jo position size determine karta hai
• 4.7.02-Risk-reward-ratio: Backtest actual R:R achieved reveal karta hai, hypothetical nahi
• 4.3.01-Support-and-resistance-levels: Technical strategies ko backtest karke validate karna zaroori hai ki woh random toh nahi hain
• 3.2.08-Market-efficiency-and-anomalies: Backtesting help karta hai identify karne mein ki anomaly real hai ya data-mined

#flashcards/stock-market

Concept Map