Reading and reproducing ML papers

The Structured Three-Pass Reading Method

Pass 1: The10-Minute Survey (Decision Pass)

Goal: Decide if this paper is worth your time.

What to read (in order):

Title + Abstract → Main claim
Introduction (first 2 paragraphs) → Problem & motivation
Section headings → Structure of solution
Conclusion → What they achieved
Figures & captions → Visual story
References → Context (recognize key citations?)

Output: Answer these questions:

What category is this? (New architecture / Training technique / Theoretical result / Dataset / Application)
What problem does it solve?
Is the claimed improvement relevant to my work?
Are the assumptions reasonable?

Pass 2: The 1-Hour Deep Dive (Understanding Pass)

Goal: Grasp the main contributions without reproducing.

How to read:

Read sequentially but skip profs (for now)
Annotate in margins: "Why this choice?", "What if we changed X?"
For each figure/table:
- Write what it claims before reading the caption
- Check if your interpretation matches
Identify the core innovation: What is the one new idea?
Note experimental setup details: Dataset, baselines, hyperparameters

Key questions to answer:

Mathematical formulation: What are the inputs, outputs, loss function?
Algorithmic innovation: What's new compared to prior work?
Experimental evidence: Is the improvement statistically significant? Ablations convincing?
Limitations: What doesn't work? What did they not test?

Formula Extracting the Core Equation

Papers often bury the key formula in prose. Extract it explicitly.

ResNet example: The paper describes residual learning as: $\mathcal{H}(\mathbf{x}) = \mathcal{F}(\mathbf{x}) + \mathbf{x}$ Why this form?

Traditional: Learn $\mathcal{H}(\mathbf{x})$ directly (mapping input to output)
Residual: Learn $\mathcal{F}(\mathbf{x}) = \mathcal{H}(\mathbf{x}) - \mathbf{x}$ (the residual)
Intuition: It's easier to learn "what to add" than "the entire transformation"
Identity shortcut: If optimal $\mathcal{H}(\mathbf{x}) = \mathbf{x}$ , network just learns $\mathcal{F}=0$

From first principles:
Gradient of loss w.r.t. input: $\frac{\partial \mathcal{L}}{\partial \mathbf{x}} = \frac{\partial \mathcal{L}}{\partial \mathcal{H}} \cdot \frac{\partial \mathcal{H}}{\partial \mathbf{x}} = \frac{\partial \mathcal{L}}{\partial \mathcal{H}} \cdot \left( \frac{\partial \mathcal{F}}{\partial \mathbf{x}} + \mathbf{I} \right)$ The $+\mathbf{I}$ term ensures gradients flow directly backward, preventing vanishing gradients.

Common mistake Common Reading Mistakes

Mistake 1: Reading linearly like a textbook
Why it feels right: We're trained to read sequentially.
The fix: Papers are not textbooks. Jump to figures/tables first. Skim methods if you know the domain.

Mistake 2: Trying to understand everything in one pass
Why it feels right: "I should understand this completely before moving on."
The fix: First pass = decision. Second pass = main ideas. Third pass (reproduction) = deep mastery. Don't get stuck on a single unclear sentence for20 minutes.

Mistake 3: Trusting reported numbers blindly
Why it feels right: "It's peer-reviewed, must be correct."
The fix: Check sample size, error bars, statistical tests. Look for ablations (did they test why it works?). ML has a reproducibility crisis—many results don't replicate.

Pass 3: The 4-8 Hour Implementation (Reproduction Pass)

Goal: Reproduce the core result from scratch.

Implementation checklist:

Environment Setup
- Note paper's framework (TensorFlow/PyTorch/JAX)
- List dependencies, hardware (GPU memory matters!)
- Find official/unofficial repos for reference (but don't peek yet!)
Data Pipeline ⚠️ Where most bugs hide
- Preprocessing steps (normalization, augmentation)
- Train/val/test splits (exactly as paper describes)
- Batch size, shuffle, number of workers
Model Architecture
- Start with smallest version (e.g., ResNet-18 before ResNet-152)
- Print shapes at every layer to catch dimension mismatches
- Initialization scheme (Xavier, Kaiming, etc.)—papers often omit this!
Training Loop
- Learning rate schedule (step decay, cosine annealing?)
- Optimizer (Adam vs SGD—huge difference!)
- Regularization (weight decay, dropout rates)
- Early stopping criteria
Verification Checkpoints
- Sanity check: Can model overfit 10 samples? (If not, bug in loss/backprop)
- Baseline check: Does it match reported baseline performance?
- Ablation check: Remove claimed innovation—does performance drop as paper claims?

Worked example Full Reproduction Walkthrough: Attention is All You Need (Transformer)

Pass 1outcome: Replaces RNs with self-attention. Claims faster training, better BLEU on translation. Pass 2 extraction:
Core innovation = Scaled Dot-Product Attention: $\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V$ Why the $\sqrt{d_k}$ scaling?
From first principles:
If $Q, K$ are $d_k$ -dimensional with entries $\sim \mathcal{N}(0,1)$ : $QK^T = \sum_{i=1}^{d_k} q_i k_i$ Variance of sum $d_k$ independent $\mathcal{N}(0,1)$ products: $\text{Var}(QK^T) = d_k \cdot \text{Var}(q_i k_i) = d_k \cdot 1 = d_k$

Without scaling, dot products grow large → softmax saturates → gradients vanish.
Dividing by $\sqrt{d_k}$ normalizes variance back to 1.

Pass 3 reproduction:

import torch
import torch.nn as nn
class ScaledDotProductAttention(nn.Module):
   def __init__(self, d_k):
       super().__init__()
       self.d_k = d_k
   
   def forward(self, Q, K, V, mask=None):
       # Q, K, V: (batch, heads, seq_len, d_k)
       scores = torch.matmul(Q, K.transpose(-2, -1)) / (self.d_k ** 0.5)
       # Why this step? Normalize variance to prevent saturation
       if mask is not None:
           scores = scores.masked_fill(mask == 0, -1e9)
           # Why? Prevent attending to padding tokens
       
       attn_weights = torch.softmax(scores, dim=-1)
       # Why softmax? Gives probability distribution over keys
       
       output = torch.matmul(atn_weights, V)
       return output, attn_weights

Verification:

✓ Overfit 10 sentence pairs → 100% train accuracy
✓ Match reported WMT'14 En-De BLEU (within 0.5 points)
✓ Ablation: Remove scaling → BLEU drops 3.2 points (paper claims 2.8)

Advanced Paper Reading Strategies

The 80/20 Focus Areas

20% of paper content gives 80% of value:

Problem statement (Intro paragraph1)
Core mathematical formulation (Usually one equation)
Main results table (Numbers that matter)
Ablation study (What actually contributes to performance)

What to skim/skip:

Related work (unless you need context)
Lengthy proofs (read only if theorem is core contribution)
Excessive experimental details (unless reproducing)

The Forecast-then-Verify Reading

Before reading methods: After Pass 1, write down:

"If I were solving this problem, I would try..."
"The main challenge is probably..."
"I predict the solution involves..."

Then read methods: Compare your forecast to their approach.

✓ Matched? → You understand the problem space well
✗ Different? → Learn why their approach is superior (or not!)

This builds researcher intuition.

Building a Paper Reading System

Organizational structure:

papers/
├── to-read/          # Pass 1 triage
├── reading/          # Pass 2 in progress
├── implemented/      # Pass 3 completed with code
└── reference/        # Key papers for quick lookup

Annotation system:

🔴 Red: Don't understand, need to revisit
🟡 Yellow: Core contribution
🟢 Green: Interesting extension idea
💡 Margin notes: "Could this apply to [my problem]?"

Reading cadence:

Daily: 1-2 Pass 1 reads (arxiv feed)
Weekly: 2-3 Pass 2 reads (deep understanding)
Monthly: 1 Pass 3 (full reproduction)

Dealing with Irreproducible Papers

Common blockers:

Missing hyperparameters
- Search author's GitHub for code
- Email authors politely: "What learning rate did you use for [experiment]?"
- Try ranges from related papers
Undisclosed dataset preprocessing
- Check official dataset papers for standard practices
- Compare reported stats (mean/std) to your pipeline
- Look for off-by-one errors in reported numbers
Cherry-picked results
- Look for error bars, multiple seeds
- If "best of 10runs" is reported, be suspicious
- Try to reproduce mean± std instead of best
Computational requirements
- Scale down: Use smaller model variant
- Use techniques from paper on different (smaller) dataset
- Focus on reproducing relative improvements not absolute numbers

When to Give Up vs. Persist

Give up if (after 2-3 solid attempts):

Core idea is poorly explained and authors don't respond
Results seem too good to be true and no one else can replicate
Paper lacks ablations—you can't isolate the contribution

Persist if:

Core idea is clear but implementation details are missing (solvable!)
Other researchers report partial success
Failure teaches you something valuable about the problem domain

Pivoting: If exact reproduction fails, try:

Reproduce on simpler/different dataset
Implement core technique in different context
Write "lessons from attempted reproduction" blog post—these are valuable!

Recall Explain Like I'm 12: Reading ML Papers

Imagine you find a LEGO instruction manual for an awesome spaceship, but:

Some pages are missing
They used special LEGO pieces you don't have
They didn't show every single step—they assumed you know basics Pass 1 = Flipping through to see if this spaceship is cool enough to build
Pass 2 = Reading carefully, figuring out the special technique they used for the wings
Pass 3 = Actually building it with your own LEGO pieces, figuring out the missing steps

Sometimes you can't build the exact same spaceship—but you learn the cool wing technique and use it in your own designs. That's reproducing papers: You're learning techniques from the smartest builders, even if you can't copy them exactly.

Why it's hard: The manual writers are experts. They forget beginers don't know stuff. They say "attach stabilizer" but don't say which stabilizer or how. You have to figure it out by trying!

Connections

Paper Writing and Publication ← Inverse process: creating reproducible papers
Experiment Tracking and Versioning ← Tools for managing reproduction attempts (MLflow, Weights & Biases)
Hyperparameter Tuning ← Missing hyperparameters is #1 reproduction blocker
Statistical Significance Testing ← Evaluating if reproduced results "match"
Model Debugging ← Techniques when your reproduction doesn't work
Transfer Learning ← Papers on ImageNet→YourTask require reproduction understanding
Attention Mechanisms ← Case study: highly-cited, heavily-reproduced architecture

#flashcards/ai-ml

What are the three passes in structured ML paper reading and their time commitments? :: Pass 1 (10 min): Survey to decide relevance | Pass 2 (1 hour): Deep dive for understanding | Pass 3 (4-8 hours): Full reproduction with code

Why does scaled dot-product attention divide by sqrt(d_k)?

Without scaling, variance of QK^T grows with d_k, causing softmax saturation and vanishing gradients. Dividing by sqrt(d_k) normalizes variance to 1.

What are the 5 verification checkpoints when reproducing a paper?

1. Overfit sanity check (10 samples) | 2. Match reported baseline | 3. Ablation check (remove innovation, see drop) | 4. Shape verification (print dimensions) | 5. Statistical match (mean ± std, not just best)

What's the difference between replication and reproduction?

Replication = exact recreation with same code/data | Reproduction = rebuild from description, implementation details may differ | Both aim to verify claims

What should you extract in Pass 2 reading?

Core mathematical formulation | Algorithmic innovation vs. prior work | Experimental setup details | Key limitations | Statistical significance of results

Why is the data pipeline the #1 source of reproduction bugs?

Papers often omit preprocessing details (normalization constants, augmentation parameters train/val splits). Small differences (like using mean=[0.5,0.5,0.5] vs. ImageNet stats) can cause large performance gaps.

What does the "Forecast-then-Verify" reading strategy involve?

Before reading methods, predict how you'd solve the problem | Then compare your approach to authors' solution | Builds researcher intuition about problem spaces

When should you give up on reproducing a paper?

After 2-3 solid attempts, if: Core idea is unclear + authors don't respond | Results are unrealistically good + no one else replicates | No ablations to isolate contribution

What are the 20% of paper sections that give 80% of value?

Problem statement (intro para 1) | Core mathematical formulation (usually one equation) | Main results table | Ablation study showing what works

What information belongs in Pass 1 annotations?

Paper category (architecture/training/theorydataset) | Problem it solves | Claimed improvement and relevance | Reasonableness of assumptions | Decision to read further or skip

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

ML papersdhna ek superpower hai, lekin thoda tricky bhi. Socho tum ek naya recipe try kar rahe ho, par instructions adhuri hain aur chef ne kuch steps assume kar liye hain ki tumhe pata honge. Research papers bhi aise hi hote hain—unme latest techniques hain, par har detail nahi hoti.

Teen-pass method use karo: Pehle 10 minute mein quickly dekho ki paper relevant hai ya nahi (title, abstract, figures). Agar haan, toh 1 ghante mein properly padho—core equation derive karo, samjho ki innovationya hai. Phir agar tumhe use karna hai, toh 4-8 ghante lagake code likho aur reproduce karo. Sabse important: verification checkpoints—pehle check karo ki model10 samples pe overfit kar sakta hai (agar nahi toh bug hai!), phir baseline numbers match karo, phir ablation test karo (innovation hata ke dekho performance girta hai ya nahi).

Ek baat yad rakho: kagaz mein jo likha hai, uska 20% tumhe 80% value dega—problem statement, main equation, results table, aur ablation study pe focus karo. Baki related work aur lambe profs ko skip kar sakte ho unless core contribution nahi samajh aa raha. Aur haan, agar reproduce nahi ho raha 2-3 baar try karne ke baad, toh authors ko politely email karo—zyada tar researchers help karte hain kyunki tum unka kaam validate kar rahe ho. Reproduction is the ultimate test: agar tumne bana liya scratch se, tabhi sach mein samajh aya!

The Structured Three-Pass Reading Method

Pass 1: The10-Minute Survey (Decision Pass)

Pass 2: The 1-Hour Deep Dive (Understanding Pass)

Pass 3: The 4-8 Hour Implementation (Reproduction Pass)

Advanced Paper Reading Strategies

The 80/20 Focus Areas

The Forecast-then-Verify Reading

Building a Paper Reading System

Dealing with Irreproducible Papers

When to Give Up vs. Persist

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Research Frontiers & Practice

Connections