Backtesting Frameworks

Question 1

(a) Gross CAGR (4 marks)

CAGR $= (V_f/V_0)^{1/n} - 1$ where $n = 3$ years.

$\text{CAGR} = \left(\frac{168000}{100000}\right)^{1/3} - 1 = 1.68^{1/3} - 1$

$1.68^{1/3} = 1.188876$, so CAGR ≈ 0.1889 = 18.89%.

• Formula (1), substitution (1), cube-root (1), answer (1).

(b) Total cost, net terminal, net CAGR (6 marks)

Cost per fill (in bps of notional):

• commission = 0.5 bps
• slippage = $k\sigma = 0.10 \times 8 = 0.8$ bps
• total = 1.3 bps = 0.00013 of notional. (2)

Cost per fill in $ = 0.00013 \times 130000 = \16.90$. Total over 1512 fills =$16.90 \times 1512 = $25{,}552.80$. (2)

Net terminal = 168000 - 25552.80 = \142{,}447.20$. Net CAGR$= (142447.2/100000)^{1/3} - 1 = 1.424472^{1/3} - 1$.$1.424472^{1/3} = 1.125215$, so net CAGR ≈ 0.1252 = 12.52%. (2)

(c) Max drawdown (3 marks)

$\text{MDD} = \frac{155000 - 126000}{155000} = \frac{29000}{155000} = 0.18710 = \textbf{18.71\%}$

• Formula peak-to-trough over peak (1), value (1), % (1).

(d) Survivorship bias (5 marks)

• Definition: using only currently surviving constituents excludes firms that were delisted, went bankrupt, or were removed for poor performance during the test window (2).
• Mechanism inflating CAGR: the excluded delisted names typically had large negative returns before removal; their absence removes losers from the sample, biasing average/compound returns upward. Also index-inclusion effect (winners added late) (1).
• Fix: source a point-in-time / as-was constituent list with full delisting history and continue holding delisted names through to their final delisting price (a database with delisted securities, e.g. CRSP-style) (2).

(e) Lump-sum vs per-trade compounding critique (6 marks)

• In the lump-sum method the full $25,552.80 is subtracted once at the end, so all costs are effectively "paid" with terminal-value dollars and do not reduce the capital base that compounds during the period (2).
• In reality each trade's cost reduces equity at the time it occurs, so subsequent compounding acts on a smaller base — the drag compounds (2).
• Therefore per-trade deduction yields a lower net terminal value and lower net CAGR than the lump-sum figure; the lump-sum method overstates performance (the error is optimistic/positive-biased) (2).

Question 2

(a) Look-ahead bias (6 marks)

• Bug: signal computed from close[t] is multiplied by ret[t] where ret[t] = (close[t]/close[t-1]-1). This uses the same bar's closing price both to decide the position and to earn that bar's return — you cannot know today's close in time to have held the position that produced today's return (3).
• Fix: lag the signal by one bar so today's return is driven by yesterday's (known) signal:

df['stratret'] = df['signal'].shift(1) * df['ret']

(3). (Accept also shifting the signal generation to use close.shift(1).)

(b) Walk-forward in/out-of-sample (7 marks)

Example 3-fold expanding (or rolling) walk-forward on 2015–2024:

(any monotone, non-overlapping, train-before-test scheme accepted) (4).

Why walk-forward over random shuffle: time-series data is autocorrelated and non-stationary; random shuffling leaks future information into training (look-ahead) and destroys temporal ordering, so it overstates out-of-sample skill. Walk-forward always trains on the past and tests on the strictly later, unseen future, matching live deployment (3).

(c) Sharpe gap (5 marks)

• The large drop (1.9 → 0.4) indicates overfitting to the in-sample data (curve-fitting to noise / too many optimized parameters) (2).
• Two fixes (any two, 1.5 each): reduce parameter count / regularize; use fewer optimization trials; enforce economic rationale; use nested cross-validation; add transaction costs during optimization; ensemble/robust parameter regions rather than a single peak.

Question 3

(a) Mean & variance of cumulative log-return (4 marks)

For sum of 252 i.i.d.: $\mu_R = 252\mu = 252(0.0006) = 0.1512$ $\sigma_R^2 = 252\sigma^2 = 252(0.012)^2 = 252(0.000144) = 0.036288,\quad \sigma_R = 0.190494$

• Mean (2), variance/sd (2).

(b) P(R<0) (5 marks)

$P(R<0) = \Phi\!\left(\frac{0-\mu_R}{\sigma_R}\right) = \Phi\!\left(\frac{-0.1512}{0.190494}\right) = \Phi(-0.79373)$ $\Phi(-0.7937) \approx 0.2137$. P(R<0) ≈ 0.2137 (≈ 21.4%).

• Standardization (2), z-value (1), CDF numeric (2).

(c) Monte Carlo function (5 marks)

import numpy as np
def mc_terminal(mu, sigma, days, n_paths):
    r = np.random.normal(mu, sigma, size=(n_paths, days))
    terminal = np.exp(r.sum(axis=1))   # start from $1, log-returns
    return terminal

(3 marks: correct shape, summing log-returns, exp to get wealth).

5th-percentile estimate: np.percentile(mc_terminal(...), 5) — sort the terminal-wealth array and take the value below which 5% of simulated outcomes fall; this is a Monte-Carlo VaR estimate of worst-case terminal wealth (2).

(d) Tail-risk critique (4 marks)

• Normal assumption has thin tails and assumes independence; real returns exhibit fat tails (excess kurtosis) and volatility clustering / autocorrelation, so extreme drawdowns occur far more often than the Normal predicts — VaR is understated (2).
• Alternative: bootstrap / block-bootstrap resampling of historical returns (or stationary bootstrap), which preserves the empirical distribution's fat tails and (with block resampling) short-range dependence (2).

[
  {"claim":"Gross CAGR of 168000/100000 over 3 yrs ~ 0.1889","code":"cagr=(Rational(168000,100000))**(Rational(1,3))-1; result=abs(float(cagr)-0.1889)<0.001"},
  {"claim":"Total cost = 1.3bps*130000*1512 = 25552.80","code":"cost=Rational(13,100000)*130000*1512; result=abs(float(cost)-25552.80)<0.01"},
  {"claim":"Net CAGR ~ 0.1252","code":"net=(Rational(1424472,1000000))**(Rational(1,3))-1; result=abs(float(net)-0.1252)<0.001"},
  {"claim":"Max drawdown = 29000/155000 ~ 0.1871","code":"mdd=Rational(29000,155000); result=abs(float(mdd)-0.1871)<0.001"},
  {"claim":"mu_R=0.1512 and sigma_R^2=0.036288","code":"muR=252*Rational(6,10000); varR=252*(Rational(12,1000))**2; result=(abs(float(muR)-0.1512)<1e-9) and (abs(float(varR)-0.036288)<1e-9)"},
  {"claim":"P(R<0)=Phi(-0.79373)~0.2137","code":"from sympy import erf,sqrt as ssqrt; z=Rational(-1512,10000)/(ssqrt(Rational(36288,1000000))); p=(1+erf(z/ssqrt(2)))/2; result=abs(float(p)-0.2137)<0.002"}
]