Understand backtesting strategies properly

Overview

Backtesting is the process of testing a trading strategy on historical data to evaluate how it would have performed in the past. It's the bridge between a trading idea and real capital deployment, but it's also where most traders fool themselves into false confidence.

What Backtesting Actually Tests

WHY We Backtest

1. Validate Logic: Does the strategy's core hypothesis (e.g., "momentum persists over 20 days") actually show up in historical data?
2. Quantify Performance: What are realistic expectations for returns, drawdowns, win rates?
3. Build Confidence: Reduces emotional doubt when the strategy hits a rough patch in live trading
4. Parameter Optimization: Find the best settings (e.g., moving average periods) within reasonable bounds

WHAT Backtesting Cannot Do

• Predict the future: Markets evolve. A strategy that worked 2010-2020 may fail2021-2026
• Account for execution reality: Slippage, liquidity, and your own emotions aren't in the backtest
• Guarantee profitability: Past performance ≠ future results (this is legally required disclosure for a reason)

The Mathematics of Backtest Reliability

Sample Size and Statistical Significance

To determine if a strategy's edge is real or luck, we need statistical significance. The required sample size depends on your desired confidence level.

Derivation from scratch:

Assume each trade is independent with win probability $p$ (unknown). You observe $n$ trades with $w$ wins. The sample win rate is $\hat{p} = w/n$.

The standard error of this proportion is: $SE = \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$

For a 95% confidence interval (1.96 standard deviations): $\text{CI} = \hat{p} \pm 1.96 \cdot SE$

If your strategy claims a 55% win rate, you want the confidence interval to not include 50% (break-even). Setting up the inequality:

$\hat{p} - 1.96\sqrt{\frac{\hat{p}(1-\hat{p})}{n}} > 0.50$

Solving for $n$ when $\hat{p} = 0.55$: $0.55 - 1.96\sqrt{\frac{0.55 \cdot 0.45}{n}} > 0.50$ $0.05 > 1.96\sqrt{\frac{0.2475}{n}}$ $0.05^2 > (1.96)^2 \cdot \frac{0.2475}{n}$ $n > \frac{3.8416 \cdot 0.2475}{0.0025} \approx 380 \text{ trades}$

WHY this matters: With only 50 trades, your 55% win rate could easily be noise. You need hundreds of trades to claim statistical confidence.

You backtest a swing trading strategy on Nifty 50 stocks (2015-2024):

• Total trades: 85
• Wins: 50 (58.8% win rate)
• Avg profit per trade: ₹2,100

Question: Is this strategy proven?

Analysis: $SE = \sqrt{\frac{0.588 \cdot 0.412}{85}} = \sqrt{\frac{0.242}{85}} = 0.053$ $\text{95\% CI} = 0.588 \pm 1.96(0.053) = [0.484, 0.692]$

The confidence interval includes 50%, meaning you cannot confidently say this strategy has an edge. You might just be seeing random variation.

WHY this step? We're checking if the observed win rate is distinguishable from coin-flip luck.

Example 2: Sufficient Sample

Same strategy, but 10 years of daily data across 200 stocks:

• Total trades: 450
• Wins: 265 (58.9% win rate)

$SE = \sqrt{\frac{0.589 \cdot 0.411}{450}} = 0.023$ $\text{95\% CI} = 0.589 \pm 1.96(0.023) = [0.544, 0.634]$

Now the interval excludes 50%—this edge is statistically significant.

Look-Ahead Bias: The Silent Killer

Common forms:

1. Using adjusted prices incorrectly: Stock splits are adjusted backward in historical data. If you backtest a rule like "buy when price < ₹100" on adjusted data, you're using future split information
2. Survivorship bias: Testing only stocks that still exist today (survivors) ignores bankrupt/delisted stocks
3. Peeking at the close: Generating a signal "when price crosses MA" but using the closing price the same day to decide—in reality, you don't know the close until after market close

Flawed backtest: "Buy Reliance when adjusted close< ₹200, sell at ₹250"

You run this on data where Reliance had a 1:1 split in 2017. The backtest shows you "bought" at ₹180 in 2015 (adjusted) and sold at ₹250 (adjusted) in 2016. Huge profit!

WHY this is wrong: In 2015, the actual price was ₹360 (before split adjustment). You never would have bought because your rule said "< ₹200" based on the real price at the time, not the backward-adjusted price.

The fix: Use point-in-time pricing or clearly separate pre/post-split analysis.

Overfitting: Torturing the Data Until It Confesses

The Mathematics of Overfitting

If you test $k$ independent strategies, each with a 5% chance of showing a "significant" result by luck (p < 0.05), the probability that at least one looks good by chance is:

$P(\text{at least one false positive}) = 1 - (0.95)^k$

Derivation:

• Probability that one test doesn't give a false positive: 0.95
• Probability that all $k$ tests don't give false positives: $(0.95)^k$
• Probability of at least one false positive: complement of the above

For $k = 20$ tests: $1 - (0.95)^{20} = 0.64$ (64% chance of fool's gold!)

You test a moving average crossover strategy:

• Fast MA: 5, 10, 15, 20, 25, 30 days (6 choices)
• Slow MA: 50, 100, 150, 200 days (4 choices)
• Stop loss: 2%, 3%, 5%, 7%, 10% (5 choices)

Total combinations: $6 \times 4 \times 5 = 120$

You find that (15, 100, 3%) gives 32% annual return with max drawdown of 8% over 2015-2020. Amazing!

WHY this is dangerous: You tested 120 strategies. By the false discovery formula: $P(\text{at least one looks good by chance}) = 1 - (0.95)^{120} \approx 0.998$

You're almost guaranteed to find something that looks good purely by luck. This is curve-fitting or data mining.

WHY this step? We're calculating the probability that our "best" result is an illusion created by testing too many variations.

Out-of-Sample Testing: The Honest Judge

WHAT this does: If performance collapses out-of-sample, your strategy was overfit.

HOW to implement:

In-sample: Develop strategy, optimize parameters
[Freeze the strategy completely]
Out-of-sample: Run once, record results, no tweaking

If you iterate back to in-sample after seeing out-of-sample results, you've contaminated the test.

More robust than single train/test split:

Method:

1. Train on 2010-2012, test on 2013 (record results)
2. Train on 2011-2013, test on 2014 (record results)
3. Train on 2012-2014, test on 2015 (record results)
4. Continue rolling forward...

Aggregate all out-of-sample test results. If the strategy degrades over time, it's not adapting to market regime changes.

WHY this step? Markets evolve. A strategy that works in one period may fail in another. Walk-forward shows you this degradation in real-time simulation.

Transaction Costs: The Returns Killer

$R_{\text{net}} = R_{\text{gross}} - \text{Costs}$

Where: $\text{Costs} = n \times (\text{Commission} + \text{Slippage} + \text{Impact})$

• $n$ = number of trades
• Commission = brokerage fees (₹20 per trade, or 0.03% of value)
• Slippage = difference between expected price and actual execution price
• Impact = price moves against you due to your own order size (relevant for large orders)

Derivation from first principles:

Your backtest shows gross annual return of 25% over 120 trades/year. You have ₹10,00,000 capital.

Assume:

• Average trade size: ₹50,000
• Commission: 0.03% each side (buy + sell) = 0.06% per round trip
• Slippage: 0.1% per side = 0.2% per round trip
• Total cost per trade: 0.26% of trade value

$\text{Cost per trade} = 0.0026 \times 50,000 = ₹130$ $\text{Annual costs} = 120 \times 130 = ₹15,600$

Your backtest shows profit of ₹2,50,000 (25% of ₹10,00,000).

$R_{\text{net}} = \frac{250,000 - 15,600}{1,000,000} = 23.44\%$

WHY this matters: High-frequency strategies with low profit per trade get destroyed by costs. A strategy with 200 bps gross profit per trade and 50 bps costs loses 25% of its edge.

Scalping strategy backtest:

• Gross profit per trade: ₹500
• Trades per day: 10
• Trading days per year: 250
• Commission + slippage per trade: ₹80

Gross annual: $500 \times 10 \times 250 = ₹12,50,000$ Costs: $80 \times 10 \times 250 = ₹2,00,000$ Net annual: $₹10,50,000$

Costs consumed 16% of gross profit. If your backtest didn't include realistic costs, you've overestimated returns by 16%.

WHY this step? We're showing that transaction costs aren't a small adjustment—they fundamentally change strategy viability, especially for high-turnover approaches.

Psychological Realism: You Are Not a Robot

Why it feels right: Numbers on a screen are emotionally neutral. 18% is just a statistic.

The Reality: When you're down 18% in real money, watching your ₹5,00,000 become ₹4,10,000, you didn't just lose statistics—you lost a new car, a vacation, your confidence. The urge to override your rules becomes overwhelming.

Steel-man: You're not stupid for feeling this way. Loss aversion is evolutionary—our ancestors who panicked at resource loss survived. The market punishes this instinct.

The Fix:

1. Trade smaller size than your backtest suggests. If backtest says 10% per position, trade 5%
2. Paper trade the strategy for 3 months before going live. Experience the emotional drawdown in real-time without money
3. Write down your rules and the backtested max drawdown. When you hit 15%, read them. Commit to following through or stopping completely—no in-between tweaking

Key Metrics to Evaluate

$\text{Sharpe} = \frac{R_p - R_f}{\sigma_p}$

• $R_p$ = portfolio return
• $R_f$ = risk-free rate (e.g., 7% for Indian government bonds)
• $\sigma_p$ = standard deviation of portfolio returns

WHY Sharpe matters: A strategy with 30% return and 40% volatility (Sharpe = 0.58) is worse than 20% return with 10% volatility (Sharpe = 1.3). You want returns per unit of risk.

Maximum Drawdown (MDD):

$\text{MDD} = \max_{t}\left(\frac{\text{Peak}_t - \text{Trough}_t}{\text{Peak}_t}\right)$

This is the largest peak-to-valley loss measured across all time points $t$. If your peak was ₹10,00,000 and the worst subsequent trough was ₹7,00,000, then $\text{MDD} = (10,00,000 - 7,00,000)/10,00,000 = 30\%$.

WHY MDD matters: This is the pain you will feel. If you can't stomach a 30% loss, don't trade this strategy.

Profit Factor:

$\text{PF} = \frac{\sum \text{Winning Trades}}{\sum |\text{Losing Trades}|}$

A PF of 1.5 means you make ₹1.50 for every ₹1 you lose. Below 1.0 means you're losing money.

Strategy results (2015-2024):

• CAGR: 22%
• Max drawdown: 25%
• Sharpe ratio: 1.1
• Profit factor: 1.8
• Win rate: 52%
• Avg win: ₹4,500
• Avg loss: ₹3,200
• Total trades: 380

Analysis:

• Sample size: 380 trades ✓ (sufficient per earlier formula)
• Sharpe 1.1: Decent risk-adjusted return
• MDD 25%: Can you handle being down ₹2,50,000 on a ₹10,00,000 account? Be honest.
• PF 1.8: Solid—nearly 2:1 gross profit to loss ratio
• Win rate 52%: Slight edge, reasonable

Verdict: Statistically valid, but your personal risk tolerance is the final judge.

Common Backtesting Mistakes

Why it feels right: Historical data providers give you adjusted prices, convenient!

The trap: You're using future knowledge (splits, dividends) that didn't exist at signal time.

Fix: Use raw prices for signals, or ensure your adjustment is strictly point-in-time.

Mistake 2: Ignoring Liquidity

Why it feels right: Your backtest shows you bought 10,000 shares of a penny stock at ₹12.

The trap: In reality, only 500 shares were available at ₹12 that day. You'd have moved the price to ₹14 filling your order.

Fix: Filter out stocks below minimum average daily volume (e.g., ₹10 lakh ADV). Model slippage conservatively.

Mistake 3: Peeking Into the Future

Why it feels right: "I'll buy when price crosses above yesterday's high, using today's close."

The trap: You don't know today's close until 3:30pm. If the cross happened at 10am, which price did you use?

Fix: Use open-to-close logic carefully. Intraday signals must use intraday data. EOD signals use previous day's data.

"Remember when you replay your old Fortnite matches to see what you did wrong? Backtesting is like that for trading. You pretend to trade on old market data to see if your strategy would've worked.

But here's the trick: just because you won a match doesn't mean you'll win the next one with the same strategy. Maybe other players were bad, or you got lucky loot.

Backtesting is the same. You have to make sure:

1. You didn't cheat by knowing the future (like replaying but already knowing where enemies spawn)
2. You played enough matches to know it wasn't just luck (one good game doesn't make you pro)
3. You account for lag and bugs (slippage and costs in trading)
4. You're honest about whether you'd actually make those moves under pressure (emotions are real)

If your backtest says you'd have made ₹10 lakh, but you tested 100 different strategies and picked the best one, you probably just found the luckiest game. That luck won't repeat."

Connections

• 4.8.01-Confirmation-bias-in-trading: Backtesting can feed confirmation bias if you cherry-pick timeframes
• 4.8.05-Overconfidence-from-past-wins: A good backtest creates dangerous overconfidence
• 4.5.03-Position-sizing-and-risk-per-trade: Backtest tells you expected drawdown, which determines position size
• 4.7.02-Risk-reward-ratio: Backtest reveals actual R:R achieved, not hypothetical
• 4.3.01-Support-and-resistance-levels: Technical strategies need backtesting to validate they're not random
• 3.2.08-Market-efficiency-and-anomalies: Backtesting helps identify if anomaly is real or data-mined

#flashcards/stock-market

Concept Map

Hinglish (regional understanding)