Reading and reproducing ML papers — Hinglish notes

6.5.1 · AI-ML › Research Frontiers & Practice

Structured Three-Pass Reading Method

Pass 1: 10-Minute Survey (Decision Pass)

Goal: Decide karo ki yeh paper tumhara time worth hai ya nahi.

Kya padhna hai (order mein):

Title + Abstract → Main claim
Introduction (pehle 2 paragraphs) → Problem aur motivation
Section headings → Solution ki structure
Conclusion → Unhone kya achieve kiya
Figures & captions → Visual story
References → Context (koi key citations pehchante ho?)

Output: In questions ka jawab do:

Yeh kis category mein aata hai? (New architecture / Training technique / Theoretical result / Dataset / Application)
Yeh kaun si problem solve karta hai?
Kya claimed improvement mere kaam se relevant hai?
Kya assumptions reasonable hain?

Pass 2: 1-Hour Deep Dive (Understanding Pass)

Goal: Reproduce kiye bina main contributions ko samjho.

Kaise padhna hai:

Sequentially padho lekin proofs skip karo (abhi ke liye)
Margins mein Annotate karo: "Yeh choice kyun?", "Agar hum X change kar dein toh?"
Har figure/table ke liye:
- Caption padhne se pehle likho ki yeh kya claim karta hai
- Check karo ki tumhari interpretation match karti hai ya nahi
Core innovation identify karo: Ek naya idea kya hai?
Experimental setup details note karo: Dataset, baselines, hyperparameters

Key questions jinke jawab dhundho:

Mathematical formulation: Inputs, outputs, loss function kya hain?
Algorithmic innovation: Prior work se kya naya hai?
Experimental evidence: Kya improvement statistically significant hai? Kya ablations convincing hain?
Limitations: Kya kaam nahi karta? Unhone kya test nahi kiya?

Common mistake Common Reading Mistakes

Mistake 1: Textbook ki tarah linearly padhna
Kyun sahi lagta hai: Hum sequentially padhne ke liye trained hain.
Fix: Papers textbooks nahi hote. Pehle figures/tables par jaao. Agar domain jaante ho toh methods skim karo.

Mistake 2: Ek hi pass mein sab kuch samajhne ki koshish karna
Kyun sahi lagta hai: "Mujhe aage badhne se pehle yeh poora samajhna chahiye."
Fix: Pehla pass = decision. Doosra pass = main ideas. Teesra pass (reproduction) = deep mastery. Kisi ek unclear sentence par 20 minute mat atko.

Mistake 3: Reported numbers par aankhein moondkar bharosa karna
Kyun sahi lagta hai: "Yeh peer-reviewed hai, sahi hoga."
Fix: Sample size, error bars, statistical tests check karo. Ablations dhundho (kya unhone test kiya ki kyun kaam karta hai?). ML mein reproducibility crisis hai—bahut se results replicate nahi hote.

Pass 3: 4-8 Hour Implementation (Reproduction Pass)

Goal: Core result ko scratch se reproduce karo.

Implementation checklist:

Environment Setup
- Paper ka framework note karo (TensorFlow/PyTorch/JAX)
- Dependencies, hardware list karo (GPU memory matter karta hai!)
- Official/unofficial repos reference ke liye dhundho (lekin abhi peek mat karo!)
Data Pipeline ⚠️ Yahaan sabse zyada bugs chhupi rehti hain
- Preprocessing steps (normalization, augmentation)
- Train/val/test splits (bilkul waise jaise paper describe karta hai)
- Batch size, shuffle, number of workers
Model Architecture
- Sabse chhote version se shuru karo (jaise ResNet-152 se pehle ResNet-18)
- Dimension mismatches pakadne ke liye har layer par shapes print karo
- Initialization scheme (Xavier, Kaiming, etc.)—papers often yeh omit kar deti hain!
Training Loop
- Learning rate schedule (step decay, cosine annealing?)
- Optimizer (Adam vs SGD—bahut bada farq hai!)
- Regularization (weight decay, dropout rates)
- Early stopping criteria
Verification Checkpoints
- Sanity check: Kya model 10 samples overfit kar sakta hai? (Agar nahi, toh loss/backprop mein bug hai)
- Baseline check: Kya yeh reported baseline performance match karta hai?
- Ablation check: Claimed innovation hatao—kya performance paper ke claim ke mutabiq girta hai?

Worked example Full Reproduction Walkthrough: Attention is All You Need (Transformer)

Pass 1 outcome: RNNs ko self-attention se replace karta hai. Translation par faster training aur better BLEU claim karta hai. Pass 2 extraction:
Core innovation = Scaled Dot-Product Attention: $Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}}) V$ $d_{k}$ scaling kyun?
First principles se:
Agar $Q, K$ $d_{k}$ -dimensional hain jiske entries $\sim N (0, 1)$ hain: $Q K^{T} = \sum_{i = 1}^{d_{k}} q_{i} k_{i}$ $d_{k}$ independent $N (0, 1)$ products ke sum ka variance: $Var (Q K^{T}) = d_{k} \cdot Var (q_{i} k_{i}) = d_{k} \cdot 1 = d_{k}$

Scaling ke bina, dot products bade ho jaate hain → softmax saturate karta hai → gradients vanish ho jaate hain.
$d_{k}$ se divide karne par variance wapas 1 ho jaata hai.

Pass 3 reproduction:

import torch
import torch.nn as nn
class ScaledDotProductAttention(nn.Module):
   def __init__(self, d_k):
       super().__init__()
       self.d_k = d_k
   
   def forward(self, Q, K, V, mask=None):
       # Q, K, V: (batch, heads, seq_len, d_k)
       scores = torch.matmul(Q, K.transpose(-2, -1)) / (self.d_k ** 0.5)
       # Yeh step kyun? Saturation rokne ke liye variance normalize karo
       if mask is not None:
           scores = scores.masked_fill(mask == 0, -1e9)
           # Kyun? Padding tokens par attend karne se rokna
       
       attn_weights = torch.softmax(scores, dim=-1)
       # Softmax kyun? Keys par probability distribution deta hai
       
       output = torch.matmul(atn_weights, V)
       return output, attn_weights

Verification:

✓ 10 sentence pairs overfit → 100% train accuracy
✓ Reported WMT'14 En-De BLEU match (0.5 points ke andar)
✓ Ablation: Scaling hatao → BLEU 3.2 points girta hai (paper 2.8 claim karta hai)

Advanced Paper Reading Strategies

80/20 Focus Areas

Paper content ka 20% value ka 80% deta hai:

Problem statement (Intro paragraph 1)
Core mathematical formulation (Usually ek equation)
Main results table (Numbers jo matter karte hain)
Ablation study (Kya actually performance mein contribute karta hai)

Kya skim/skip karna hai:

Related work (jab tak context ki zaroorat na ho)
Lengthy proofs (sirf tabhi padho jab theorem core contribution ho)
Excessive experimental details (jab tak reproduce na kar rahe ho)

Forecast-then-Verify Reading

Methods padhne se pehle: Pass 1 ke baad likho:

"Agar main yeh problem solve karta, toh main try karta..."
"Main challenge shayad yeh hai..."
"Mera prediction hai ki solution mein yeh shaamil hoga..."

Phir methods padho: Apna forecast unke approach se compare karo.

✓ Match kiya? → Tum problem space ko achhi tarah samajhte ho
✗ Alag? → Seekho ki kyun unka approach behtar hai (ya nahi hai!)

Yeh researcher intuition build karta hai.

Paper Reading System Banana

Organizational structure:

papers/
├── to-read/          # Pass 1 triage
├── reading/          # Pass 2 in progress
├── implemented/      # Pass 3 completed with code
└── reference/        # Key papers for quick lookup

Annotation system:

🔴 Red: Samajh nahi aaya, dobara dekhna hai
🟡 Yellow: Core contribution
🟢 Green: Interesting extension idea
💡 Margin notes: "Kya yeh [meri problem] par apply ho sakta hai?"

Reading cadence:

Daily: 1-2 Pass 1 reads (arxiv feed)
Weekly: 2-3 Pass 2 reads (deep understanding)
Monthly: 1 Pass 3 (full reproduction)

Irreproducible Papers se Deal Karna

Common blockers:

Missing hyperparameters
- Author ka GitHub code ke liye search karo
- Authors ko politely email karo: "Aapne [experiment] ke liye kaun sa learning rate use kiya?"
- Related papers se ranges try karo
Undisclosed dataset preprocessing
- Standard practices ke liye official dataset papers check karo
- Reported stats (mean/std) ko apni pipeline se compare karo
- Reported numbers mein off-by-one errors dhundho
Cherry-picked results
- Error bars, multiple seeds dhundho
- Agar "best of 10 runs" reported hai, toh suspicious raho
- Best ki jagah mean ± std reproduce karne ki koshish karo
Computational requirements
- Scale down: Chhota model variant use karo
- Paper ki techniques ko alag (chhote) dataset par use karo
- Absolute numbers ki jagah relative improvements reproduce karne par focus karo

Kab Chhodna hai vs. Lagey Rehna

Chhod do agar (2-3 solid attempts ke baad):

Core idea poorly explained hai aur authors respond nahi karte
Results bahut zyada achhe lagte hain aur koi aur replicate nahi kar pa raha
Paper mein ablations nahi hain—tum contribution isolate nahi kar sakte

Lage raho agar:

Core idea clear hai lekin implementation details missing hain (solvable!)
Doosre researchers partial success report karte hain
Failure tumhe problem domain ke baare mein kuch valuable sikhaati hai

Pivoting: Agar exact reproduction fail ho jaaye, try karo:

Simpler/alag dataset par reproduce karo
Core technique ko alag context mein implement karo
"Attempted reproduction se lessons" blog post likho—yeh valuable hote hain!

Recall Explain Like I'm 12: ML Papers Padhna

Soch lo tumhe ek awesome spaceship ke liye LEGO instruction manual mila, lekin:

Kuch pages missing hain
Unhone special LEGO pieces use kiye jo tumhare paas nahi hain
Unhone har ek step nahi dikhaya—unhone assume kiya ki tum basics jaante ho Pass 1 = Dekhne ke liye flip karo ki yeh spaceship banane laayak cool hai ya nahi
Pass 2 = Dhyan se padhna, wings ke liye jo special technique unhone use ki wo samajhna
Pass 3 = Actually apne LEGO pieces se banao, missing steps figure out karo

Kabhi kabhi tum exact wahi spaceship nahi bana paate—lekin tum cool wing technique seekh jaate ho aur ise apne designs mein use karte ho. Yahi hai papers reproduce karna: Tum sabse smart builders se techniques seekh rahe ho, chahe unhe exactly copy na kar pao.

Kyun mushkil hai: Manual writers experts hain. Woh bhool jaate hain ki beginners cheezein nahi jaante. Woh kehte hain "stabilizer attach karo" lekin nahi batate kaun sa stabilizer ya kaise. Tumhe try karke figure out karna padta hai!

Connections

Paper Writing and Publication ← Ulta process: reproducible papers banana
Experiment Tracking and Versioning ← Reproduction attempts manage karne ke tools (MLflow, Weights & Biases)
Hyperparameter Tuning ← Missing hyperparameters #1 reproduction blocker hai
Statistical Significance Testing ← Evaluate karo ki reproduced results "match" karte hain ya nahi
Model Debugging ← Techniques jab tumhara reproduction kaam na kare
Transfer Learning ← ImageNet→YourTask papers ko reproduce karne ki understanding chahiye
Attention Mechanisms ← Case study: highly-cited, heavily-reproduced architecture

#flashcards/ai-ml

Structured ML paper reading mein teen passes kya hain aur unka time commitment kya hai? :: Pass 1 (10 min): Relevance decide karne ke liye Survey | Pass 2 (1 ghanta): Samajhne ke liye Deep dive | Pass 3 (4-8 ghante): Code ke saath Full reproduction

Scaled dot-product attention sqrt(d_k) se kyun divide karta hai?

Scaling ke bina, QK^T ka variance d_k ke saath badhta hai, jisse softmax saturation aur vanishing gradients hote hain. sqrt(d_k) se divide karne par variance 1 ho jaata hai.

Paper reproduce karte waqt 5 verification checkpoints kya hain?

1. Overfit sanity check (10 samples) | 2. Reported baseline match karo | 3. Ablation check (innovation hatao, drop dekho) | 4. Shape verification (dimensions print karo) | 5. Statistical match (mean ± std, sirf best nahi)

Replication aur reproduction mein kya farq hai?

Replication = same code/data se exact recreation | Reproduction = description se rebuild, implementation details alag ho sakti hain | Dono claims verify karne ki koshish karte hain

Pass 2 reading mein kya extract karna chahiye?

Core mathematical formulation | Prior work se algorithmic innovation | Experimental setup details | Key limitations | Results ki statistical significance

Data pipeline reproduction bugs ka #1 source kyun hai?

Papers often preprocessing details omit kar deti hain (normalization constants, augmentation parameters, train/val splits). Chhote farq (jaise mean=[0.5,0.5,0.5] vs. ImageNet stats use karna) se bade performance gaps aa sakte hain.

"Forecast-then-Verify" reading strategy mein kya hota hai?

Methods padhne se pehle, predict karo ki tum problem kaise solve karoge | Phir apna approach authors ke solution se compare karo | Problem spaces ke baare mein researcher intuition build karta hai

Paper reproduce karna kab band karna chahiye?

2-3 solid attempts ke baad, agar: Core idea unclear hai + authors respond nahi karte | Results unrealistically good hain + koi aur replicate nahi kar pa raha | Contribution isolate karne ke liye ablations nahi hain

Paper ke 20% sections kaun se hain jo 80% value dete hain?

Problem statement (intro para 1) | Core mathematical formulation (usually ek equation) | Main results table | Ablation study jo dikhaye kya kaam karta hai

Pass 1 annotations mein kaun si information honi chahiye?

Paper category (architecture/training/theory/dataset) | Jo problem yeh solve karta hai | Claimed improvement aur relevance | Assumptions ki reasonableness | Aage padhne ya skip karne ka decision