Learn about clean historical data sourcing — Hinglish notes

6.2.2 · Stock-Market › Backtesting Frameworks

Intuition Clean data kyun backtesting ki neenv hai

Socho agar tum ek car ki fuel efficiency test karo sirf downhill roads par drive karke—tumhe amazing results milenge jo real world mein kaam nahi karte. Dirty historical data par backtesting bilkul yahi karti hai: woh tumhe aisi profits dikhati hai jo kabhi thi hi nahi, kyunki data mein hidden biases hain. Clean data sourcing ka matlab hai aisi price/volume records lena jo reflect karti hain woh jo tum us waqt actually dekh sakte the aur trade kar sakte the, na ki baad mein cleanup ki gayi data. Teen silent killers: survivorship bias (sirf winners tumhare dataset mein rehte hain), look-ahead bias (future information ka use karna), aur corporate action errors (splits/dividends ka galat calculation). Ek strategy jo 40% annually return karti lagti hai woh actually paise gawa sakti hai jab reality-aligned data par test ki jaaye.

Historical data "dirty" kya banata hai?

Historical stock data char main channels se corrupt hoti hai:

1. Survivorship Bias

Aisa kyun hota hai:

Free data providers aksar sirf currently-listed tickers maintain karte hain
Har saal ~5-8% stocks delist hoti hain (bankruptcy, acquisitions, compliance failures)
2000-2020 ka ek backtest jo dotcom busts ya 2008 failures miss karta hai woh kabhi un losses ko nahi dekhega

Mathematical impact:

Chalte hain bias magnitude derive karte hain. Maano:

True population: $N_{0}$ stocks at time $t = 0$
Survival rate: $s$ per year
Tumhare dataset mein sirf survivors hain: $N_{s} = N_{0} \cdot s^{T}$ after $T$ years

Agar average return $μ$ hai survivors ke liye aur $μ_{d e a d} = - 0.8$ (80% loss) delistings ke liye:

$True return = s \cdot μ + (1 - s) \cdot μ_{d e a d}$

$Biased return = μ$

$Bias = μ - [s \cdot μ + (1 - s) \cdot μ_{d e a d}] = (1 - s) (μ - μ_{d e a d})$

$s = 0.94$ (6% annual delisting), $μ = 0.12$ , $μ_{d e a d} = - 0.8$ ke saath:

$Bias = 0.06 \times (0.12 - (- 0.8)) = 0.06 \times 0.92 = 5.5% per year$

Tumhara backtest 5.5 percentage points annually inflate hua hai.

2. Look-Ahead Bias

Yeh kaise chupke se aa jaata hai:

Price data usually splits/dividends ke liye adjusted provide ki jaati hai. 2020-01-15 par, tum dekhte ho:

Adjusted price: $100
Yeh 2020-06-01 par ek 2-for-1 split reflect karta hai

Trap: Agar tumhari strategy 2020-01-15 par " $100" c a l c u l a t i o n s m e in u se k a r t ihai y e h soc h t e h u e k ii s k ama tl ab " s ha r e p r i ce " hai, t u mim pl i c i tl y p os t - s pl i t in f or ma t i o n u se k a r r ah e h o . * * U na d j u s t e d * * p r i ce$ 200 tha.

Sahi usage: Returns ke liye hamesha adjusted prices use karo, absolute price levels ke liye unadjusted (jaise "buy stocks under $10").

3. Corporate Action Errors

Types aur adjustments:

| Action | Effect | Adjustment | |-----|------------| | Split $n : 1$ | Shares $n$ se multiply hote hain | Saare prior prices $n$ se divide karo | | Dividend $D$ | Cash pay out hoti hai | Prior prices se $D$ subtract karo | | Merger | Ticker change hota hai | Histories ko correct ratio ke saath link karo | | Spinoff | Nayi entity create hoti hai | Parent value proportionally split karo |

Dividend adjustment ki derivation:

Dividend se pehle ka din: stock ki worth $P$ , isme $D$ cash value shamil hai. $P = P_{business} + D$

Baad ka din (ex-dividend): cash chali gayi, business raha: $P_{ex} = P_{business} = P - D$

Price $D$ se "drop" hoti hai, lekin shareholder value preserve rehti hai (stock + cash = constant). Adjusted series mein koi drop nahi dikhna chahiye:

$P_{adj, before} = P_{raw, before} \times \frac{P _{ex}}{P} = P_{raw, before} \times \frac{P - D}{P}$

4. Point-in-Time Data

Financial data retroactively kyun change hoti hai:

Restatements: Company accounting error discover karti hai, 3 saal ki earnings revise karta hai
Reclassifications: Sector assignments change hote hain (kya Tesla automotive hai ya tech?)
Filing delays: 10-K fiscal year-end ke 90 din baad due hoti hai, lekin bahut log late file karte hain

Tumhe kya chahiye:

2015-03-15 par backtesting ke liye:

Financials use karo jo 2015-03-15 tak file ho chuki thi, 2018 ki "final restated" version nahi
Sector classifications use karo jo 2015-03-15 tak ki thi
Agar 10-K abhi file nahi hui thi, tumhare paas woh data nahi hai

Iske bina: Tum future knowledge (restatement) se trade kar rahe ho, performance inflate ho rahi hai.

Common mistake "Cleaned data ka matlab gaps fill karna hai"

Galat intuition: "Main sirf missing prices forward-fill kar dunga ya next available value use kar lunga."

Yeh sahi kyun lagta hai: Data mein gaps irritating hain; unhe fill karna tumhara code cleaner banata hai aur crashes rokta hai.

Trap:

Ek missing price ka matlab ho sakta hai stock halted thi (news pending) ya illiquid thi
Forward-filling dikhata hai tumhara kal ki price par "holding" hai, lekin tum wahan trade nahi kar sakte the
Tumhara backtest assume karta hai tum ek aisi price par exit kiye jo exist hi nahi karti thi

Fix:

Missing data explicitly mark karo (NaN)
Jindinon data nahi hai unpar trading signals skip karo
Ya last-available price use karo lekin position ko illiquid flag karo (wider slippage apply karo)

Real example: 2008 crisis ke dauran, bahut saare corporate bonds ke quotes din bhar ke liye nahi the. Forward-filling dikhata hai tumhara bond portfolio old prices par stable hai jabki reality mein 30% crashes the. Tumhara backtest dikhata hai tum pre-crash prices par exit kiye = pure fiction.

Clean data requirements checklist

Data source tiers

Tier 1: Professional (Mehanga lekin Clean)

Bloomberg Terminal: $24,000/year, point-in-time fundamentals, delisting returns
Refinitiv (Reuters): Similar pricing, global coverage
FactSet: Institutional-grade, strong corporate actions

Pros: Survivorship-free, accurate splits/dividends, point-in-time toggle Cons: Individuals ke liye cost prohibitive

Tier 2: Specialized Backtest Vendors

Norgate Data: $300-800/year, US equities with delisted stocks, accurate adjustments
CSI Data: Similar, futures include karta hai
Sharadar (Quandl/Nasdaq Data Link): $50-200/month, achhe fundamentals, delisting coverage

Pros: Backtesting ke liye bana hua, documented methodology, affordable Cons: Limited asset classes, mostly US-focused

Tier 3: Free/API (Heavy Cleaning Chahiye)

Yahoo Finance: Free, survivorship bias, spotty delisting data, occasional errors
Alpha Vantage: Free tier limited, Yahoo se better
IEX Cloud: Cheap API, current data focus

Pros: Accessible, seekhne ke liye achha Cons: Manual survivorship correction chahiye, har corporate action verify karo, koi point-in-time fundamentals nahi

Worked example Yahoo Finance data cleaning

Task: Free sources se ek survivorship-bias-free dataset banana.

Step 1 – Delisted tickers lo:

# Yahoo delistings list nahi karta; external source use karo
# Wikipedia se historical S&P 500 changes download karo
# Ya NASDAQ delisting lists scrape karo

Yeh step kyun: Free data mein sirf survivors hain; tumhe manually dead stocks add karne hain.

Step 2 – Splits download aur verify karo:

import yfinance as yf
data = yf.download('AAPL', start='2010-01-01', actions=True)
# 'Stock Splits' column check karo
# Apple investor relations website se verify karo

Yeh step kyun: Yahoo ki split data mein errors hain (~2% tickers mein). Major names cross-check karo.

Step 3 – Custom adjustments calculate karo:

# Agar Yahoo ka adjusted close galat lagta hai, recompute karo:
adj_factor = 1.0
for date in reversed(dates):
    if split_on[date]:
        adj_factor /= split_ratio[date]
    if dividend_on[date]:
        adj_factor *= (price[date] - dividend[date]) / price[date]
    adjusted_prices[date] = raw_prices[date] * adj_factor

Yeh step kyun: Tum adjustment logic control karte ho, errors pakad sakte ho.

Result: Usable lekin imperfect data. Professional sources ke mukable ~1-2% residual errors expect karo.

Data validation techniques

Worked example Automated data quality pipeline

def validate_ohlc(df):
    """Check OHLC integrity."""
    violations = (
        (df['Low'] > df['Open']) |
        (df['Low'] > df['Close']) |
        (df['High'] < df['Open']) |
        (df['High'] < df['Close'])
    )
    return df[violations]  # Returns bad rows
def detect_split_errors(df, threshold=0.4):
    """Find unadjusted splits (40%+ overnight moves)."""
    returns = df['Close'].pct_change()
    suspect = returns[abs(returns) > threshold]
    return suspect
# Har ticker par run karo
bad_ohlc = validate_ohlc(price_data)
split_issues = detect_split_errors(price_data)

Yeh step kyun: 90% data errors backtest kharaab karne se pehle pakad leta hai. Yeh 10-minute validation weeks of debugging false signals bachata hai.

Recall Ek 12-saal ke bachche ko explain karo

Socho tum test kar rahe ho ki tumhara naya skateboard trick kaam karta hai ya nahi, sirf un bacchon ke videos dekh kar jo succeed kiye aur X Games tak pahunche. Un bacchon ke saare videos jo try kiya aur fail kiya tumhare collection mein nahi hain kyunki unhone skating chhod di. Agar tum sirf success videos dekh kar trick seekhte ho, tumhe lagega yeh bahut easy hai—tumne kabhi toote haath aur quit karne ki koshishein nahi dekhi.

Clean historical data waise hai jaise sabhi logon ke videos milein jo trick try kiya, failures including. Stock backtesting mein:

Dirty data = sirf wahi companies jo survive ki (X Games winners)
Clean data = saari companies including bankruptcies (jinlogo ne try kiya sabhi)

Kuch videos baad mein edit ki gayi hain bhi—shayad unhone woh part cut kar diya jahan bachcha pehle gira tha. Yahi look-ahead bias hai: video aisi info dikhati hai jo film hone par obvious nahi thi. Clean data ka matlab hai sab kuch exactly waise dekhna jaise hua tha, koi edited-in future knowledge nahi.

Connections

6.2.01-Define-backtesting-vs-paper-trading – Backtesting mein clean data kyun zyada matter karta hai
6.2.03-Understand-transaction-cost-modeling – Clean data mein realistic fills shamil hone chahiye
6.1.02-Understand-compounding-vs-averagereturns – Survivorship bias compound returns inflate karta hai
5.3.01-Learn-about-the-efficient-market-hypothesis – Point-in-time data true EMH test karta hai
6.4.01-Identify-overfitting-in-strategy-parameters – Dirty data false parameter optima create karta hai

#flashcards/stock-market

Historical data mein survivorship bias kya hai? :: Jab tumhare dataset mein sirf wahi securities hain jo present tak survive karti rahin, delisted/bankrupt companies ko exclude karke, ek upward-biased performance record create hota hai kyunki failures invisible ho jaati hain.

Survivorship bias backtest returns kyun inflate karta hai?

Kyunki yeh bankruptcies/delistings se saare losing trades remove kar deta hai (typically -80% to -100% losses), sirf winners bacha kar. ~6% annual delistings aur -80% par losers ke saath, yeh ~5.5% fake annual return add karta hai.

Look-ahead bias kya hai?

Backtesting mein aisi information use karna jo historical decision time par available nahi thi—jaise restated financials, future index memberships, ya adjusted prices jo price-level rules ke liye incorrectly use ki gayi hain.

Adjusted vs. unadjusted prices kaise use karne chahiye?

Return calculations ke liye adjusted prices use karo (woh splits/dividends account karte hain), lekin absolute price rules ke liye unadjusted prices (jaise "buy stocks under $10") future corporate actions se look-ahead bias avoid karne ke liye.

Point-in-time data kya hai?

Historical data jo information ko exactly waise reflect karta hai jaisa ek specific date par jaana jaata tha, baad mein restate ki gayi jaankari nahi—fundamentals ke liye critical jo aksar initial filing ke saalon baad revise hoti hain.

Missing data forward-fill karna bias kyun create karta hai?

Ek missing price ka matlab ho sakta hai stock halted ya illiquid thi—tum wahan trade nahi kar sakte the. Forward-filling assume karta hai tum ek aisi price par exit kiye jo exist nahi karti thi, crises ke dauran khaaskar khatarnak jab quotes din bhar ke liye gayab ho jaate hain.

OHLC data integrity validate karne ke liye char checks kaun se hain?

1) Low ≤ Open aur Close ≤ High, 2) High ≥ Low, 3) Price changes mein volume > 0 hona chahiye, 4) Koi duplicate timestamps nahi. Violations corrupted data indicate karte hain.

Dividend ke liye prices backward kaise adjust karte hain?

Saare prior prices ko (P - D)/P se multiply karo jahan P dividend se pehle ki price hai aur D dividend amount hai, proportional return preserve karte hue cash drop remove karke.

Tier 1 aur Tier 2 data sources mein kya fark hai?

Tier 1 (Bloomberg, Refinitiv)

20 k + / y e a r p r o f ess i o na l t er mina l s hain f u l l p o in t - in - t im e a u r d e l i s t in g co v er a g e k es aa t h . T i er 2 (N or g a t e, S ha r a d a r)

300-800/year backtest-focused vendors hain survivorship-free data ke saath lekin kam asset class coverage ke saath.

Corporate action adjustments reverse chronological order mein kyun apply karni chahiye?

Kyunki har adjustment factor post-action price par depend karta hai. Aaj se shuru karke aur backward kaam karte hue, tum har factor se multiply karte ho (splits, dividends) yeh reconstruct karne ke liye ki historical prices aaj ke share structure mein "kya honi chahiye thi."