Reading and reproducing ML papers: Worked examples

Before any symbol appears, a promise: every quantity that shows up in a formula here will first be named in plain words. If you see $d_{k}$ , you will already know it means "how many numbers are in one query vector". Nothing is assumed.

The scenario matrix

When you reproduce a paper, the difficulties fall into a handful of repeating cell classes. Think of this table as a checklist of "have I met this monster before?"

Cell	Class of situation	What makes it tricky	Example that covers it
A	Normal happy path	Everything stated, you just rebuild it	Ex 1
B	Missing detail (paper omits init / LR)	You must infer an unstated choice	Ex 2
C	Degenerate input — zero / empty / length-1	Formula must not divide by zero or crash	Ex 3
D	Sign / direction case — gradient can grow OR vanish	Same formula, opposite behaviour	Ex 4
E	Limiting behaviour — a quantity → 0 or → ∞	What happens at the extreme	Ex 5
F	Numbers don't match — you're off from the paper	Is it a bug or noise?	Ex 6
G	Real-world word problem — "should I even reproduce?"	Cost/benefit decision, not maths	Ex 7
H	Exam-style twist — a subtle trap in the formula	The "gotcha" they test you on	Ex 8

Each example below is tagged with its cell. Together they touch every row.

Recall Which cell is the scariest and why?

Cell F (numbers don't match) ::: because you cannot tell from the number alone whether it is a real bug or ordinary random variation — you need a statistical answer, not a guess.

Cell A — The happy path

Look at the figure below: it traces the shapes through the pipeline like water through pipes. Even without seeing the image, here is what it shows in words — a $Q, K, V$ box of shape $4 \times 8$ flows right into a $Q K^{⊤}$ box of shape $4 \times 4$ (pair scores), then into a scale-and-softmax box still $4 \times 4$ (row weights), then into a final $\times V$ box that returns to $4 \times 8$ — the same shape it started with.

Figure — Reading and reproducing ML papers

Worked example

Steps.

Compute $Q K^{⊤}$ . Why this step? $K^{⊤}$ (the transpose) flips $K$ from $4 \times 8$ to $8 \times 4$ , so the shared "8" lines up and can be summed away. A $(4 \times 8) \cdot (8 \times 4)$ multiply gives a $4 \times 4$ grid — one similarity score for every pair of tokens.
Divide by $d_{k} = 8 \approx 2.828$ . Why? This is the variance-taming step from the parent note; it changes the values, never the shape. Still $4 \times 4$ .
Apply softmax across each row. Why? Softmax turns each row of raw scores into a set of weights that sum to 1 — "how much does token $i$ pay attention to each token". Shape unchanged: $4 \times 4$ .
Multiply by $V$ ( $4 \times 8$ ). Why? $(4 \times 4) \cdot (4 \times 8)$ : the shared "4" cancels, leaving $4 \times 8$ — a new 8-number description for each of the 4 tokens.

Answer: $4 \times 8$ — same shape as we started with. Attention rewrites each token, it does not change how many there are or how long each is.

Verify: matrix shape rule $(m \times n) (n \times p) = (m \times p)$ applied twice: $(4 \times 8) (8 \times 4) = 4 \times 4$ , then $(4 \times 4) (4 \times 8) = 4 \times 8$ . ✓

Cell B — The missing detail

Worked example

Steps.

Recall what initialization means: before training, every weight is a random number. Why care? If those random numbers are too big, signals explode; too small, they vanish — training stalls before it starts.
Note the activation: ReLU (outputs the input if positive, else 0). Why does this matter? ReLU throws away half the signal on average, so a scheme built for symmetric activations under-supplies variance.
Choose Kaiming (He) initialization, whose variance is $\frac{2}{n _{in}}$ where $n_{in}$ is the number of inputs to a neuron. Why the factor 2? It exactly compensates for the "half the signal killed by ReLU" effect. (Xavier init uses $\frac{1}{n _{in}}$ and is right for tanh, wrong for ReLU.)
Sanity-scale check for a layer with $n_{in} = 512$ : standard deviation $= 2/512 \approx 0.0625$ .

Answer: switch from the (wrongly assumed) Xavier to Kaiming init, std $\approx 0.0625$ for a 512-input layer. See Model Debugging for how to confirm this closes the gap.

Verify: $2/512 = 0.00390625 = 0.0625$ exactly. ✓

Lesson for the matrix: when a paper is silent, the default that matches the activation is almost always the intended choice.

Cell C — Degenerate input

Worked example

Case (i): the empty (zero-length) sequence.

$Q$ , $K$ , $V$ are each $0 \times 8$ — a grid with no rows at all. Compute $Q K^{⊤}$ : $(0 \times 8) (8 \times 0) = 0 \times 0$ . Why flag this? There is no row to apply softmax to.
Softmax over an empty row would compute $\frac{e ^{(nothing)}}{\sum (nothing)}$ — a sum over zero terms $= 0$ , so we divide by zero. Why this is the true crash case: unlike length-1, the empty sequence produces a genuine $0/0$ . Real frameworks either return an empty $0 \times 8$ tensor or raise an error, so you must guard against empty batches explicitly (filter them out, or skip the block). This is the degenerate cell's real teeth.

Case (ii): the length-1 sequence.

$Q$ , $K$ , $V$ are each $1 \times 8$ . Compute $Q K^{⊤}$ : $(1 \times 8) (8 \times 1) = 1 \times 1$ — a single number, the token's similarity with itself.
Divide by $8$ . Why still safe? $d_{k}$ depends on $d_{k} = 8$ , not on sequence length, so there is no division by zero even though the sequence is minimal.
Softmax over a row of length 1. Why is this the crux? Softmax of a single value $s$ is $\frac{e ^{s}}{e ^{s}} = 1$ — always exactly 1, regardless of $s$ : the denominator is a sum over one term, never zero.
Multiply the weight $1$ by $V$ ( $1 \times 8$ ): output $= V$ unchanged.

Answer: length-1 is perfectly well-defined (the lone token attends to itself with weight $1.0$ , output $= V$ ); the empty sequence is the dangerous case — a $0/0$ softmax — so guard against zero-length inputs before the attention block. Run both as sanity checks.

Verify: softmax of a single element $s$ : $e^{s} / e^{s} = 1$ for every $s$ ; and the empty-sequence softmax denominator is a sum over zero terms $= 0$ . ✓

Cell D — The sign / direction case

Worked example Ex 4 (Cell D) · Residual connection: does the gradient grow, vanish, or exactly cancel?

The parent note gave the residual gradient $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial H} (\frac{\partial F}{\partial x} + I) .$ In plain words: $L$ is the loss (how wrong we are), $x$ is a layer's input, $H$ is the block output — the full thing the residual block produces, $H (x) = F (x) + x$ , so $\partial L / \partial H$ just means "how the loss changes when the block's output changes"; $F$ is the learned "residual" part (what the layer adds); and $I$ is the identity (leave-it-alone) term. Suppose in one layer the learned part contributes a gradient factor $\partial F / \partial x$ of $0.1$ (near-vanishing), in another it contributes $- 0.9$ , and in a third — the nasty edge case — it contributes exactly $- 1$ .

Forecast: which is more dangerous — a small positive residual gradient, a negative one, or an exact $- 1$ ?

Look at the figure below: two "gradient highways", one with a weak on-ramp, one with a subtractive ramp. In words, it shows the learned path ( $+ 0.1$ or $- 0.9$ ) running alongside the identity path ( $+ I$ ), and the two adding to give $1.1$ and $0.1$ respectively — both non-zero.

Worked example

Steps.

Case $+ 0.1$ : the bracket is $0.1 + 1 = 1.1$ . Why does this matter? Even though the learned path is almost dead, the $+ I$ keeps the total at $1.1$ — the gradient still flows. This is exactly the vanishing-gradient rescue.
Case $- 0.9$ : the bracket is $- 0.9 + 1 = 0.1$ . Why check the negative sign? A residual path can work against the identity. Here it nearly cancels it, leaving only $0.1$ — the gradient shrinks, but critically it is still non-zero and finite.
Edge case $- 1$ : the bracket is $- 1 + 1 = 0$ . Why call it out? This is the one value where a single residual block does zero out the gradient — the learned part exactly opposes the identity. In practice $\partial F / \partial x$ landing on exactly $- 1$ is vanishingly unlikely and never persists across many layers/dimensions, which is why residual nets are safe in aggregate — but you should know the failure point exists.
Compare to a plain (non-residual) layer where the bracket would just be $0.1$ or $- 0.9$ with no $+ 1$ : over 50 stacked layers, $0. 1^{50}$ is astronomically tiny — dead. With $+ 1$ , each layer is near 1, so the product stays healthy.

Answer: the $+ I$ term dominates whenever the learned part is small (case $+ 0.1 \to 1.1$ ), buffers even an adversarial part (case $- 0.9 \to 0.1$ ), and vanishes only at the exact knife-edge $\partial F / \partial x = - 1$ (case $- 1 \to 0$ ). See Attention Mechanisms and Transfer Learning where the same "keep a direct path" trick recurs.

Verify: $0.1 + 1 = 1.1$ , $- 0.9 + 1 = 0.1$ (both non-zero), and $- 1 + 1 = 0$ (the exact cancel). ✓

Cell E — Limiting behaviour

Look at the figure below: it plots two curves against $d_{k}$ . In words — the unscaled spread $d_{k}$ climbs steeply ( $1.4$ at $d_{k} = 2$ up past $11$ at $d_{k} = 128$ ), while the scaled spread sits flat on the line $1$ for every $d_{k}$ .

Worked example

Steps.

Unscaled spread (standard deviation) $= d_{k}$ . For $d_{k} = 2$ : $2 \approx 1.41$ . For $d_{k} = 128$ : $128 \approx 11.3$ . Why standard deviation not variance? Std is in the same units as the scores themselves, so it directly tells us how large the scores fed into softmax get.
As $d_{k} \to \infty$ , $d_{k} \to \infty$ : scores become enormous. Why is that bad? Softmax of very large gaps sends almost all weight to a single token — the distribution collapses to a hard one-hot, and its gradient becomes almost zero (saturation). Learning freezes.
Scaled version: divide scores by $d_{k}$ , so the spread becomes $d_{k} / d_{k} = 1$ for every $d_{k}$ . Why this exact divisor? It is the one number that cancels the growth, holding the spread constant no matter how big the model gets.

Answer: unscaled, the limit $d_{k} \to \infty$ drives softmax into total saturation (flat gradients); the $d_{k}$ divide pins the spread at 1 forever. That is why the scaling exists.

Verify: $128 \approx 11.31$ and $128 / 128 = 1$ . ✓

Cell F — Numbers don't match

Worked example

Steps.

Compute your mean: $(26.7 + 27.1 + 26.8 + 27.2 + 26.9) /5 = 134.7/5 = 26.94$ . Why the mean? A single run is a coin flip; the mean estimates your method's true score.
Compute the spread (sample standard deviation). Deviations from 26.94: $- 0.24, 0.16, - 0.14, 0.26, - 0.04$ . Squared: $0.0576, 0.0256, 0.0196, 0.0676, 0.0016$ , sum $= 0.172$ . Divide by $n - 1 = 4$ : $0.043$ ; square root $\approx 0.2074$ . Why $n - 1$ ? Using $n - 1$ (not $n$ ) corrects the slight under-estimate you get from measuring spread around your own sample mean.
How many standard deviations away is the paper's 27.3? $(27.3 - 26.94) /0.2074 \approx 1.74$ . Why this ratio? It is a "z-score" — a distance measured in units of your own noise. Under 2 is the usual "plausibly the same" threshold.

Answer: the paper's 27.3 is about 1.74 standard deviations above your mean — inside the normal seed-to-seed wobble, so this is noise, not a bug. Do a proper significance test before declaring failure, and log every seed with Experiment Tracking and Versioning.

Verify: mean $= 26.94$ ; sample std $\approx 0.2074$ ; z $\approx 1.736$ , which is $< 2$ . ✓

Cell G — The real-world word problem

Worked example

Steps.

Separate the claim from the tool. Why? Pass-1 reading (parent note) asks "is the improvement relevant to my work?" Here the 0.3% headline is not — but the augmentation trick might be a reusable building block, independent of the headline benchmark.
Estimate value-per-hour of each part. Why? Time is your scarcest resource; you rank tasks by (value gained) ÷ (cost). Full reproduction of the headline: high cost (40h + 8 GPU-days), near-zero product value → a terrible ratio. The augmentation trick alone: perhaps 4 hours to lift into your pipeline, reusable across many future products → a strong ratio.
Apply the 80/20 rule from the parent note. Why? A small slice of a paper (here, one augmentation routine) usually carries most of the reusable value; chasing the full benchmark is the low-value 80% of effort.
Decide and log the decision. Why log it? So future-you (and teammates) know the reproduction was skipped on purpose, not forgotten — record it in Experiment Tracking and Versioning.

Answer: don't reproduce the headline result; extract and unit-test only the augmentation trick (~4h), skip the full-benchmark chase. This is a cost/benefit call, not a maths problem — and it is the most common real decision. When you later have your own results worth reporting, see Paper Writing and Publication.

Verify: decision rule "reproduce a component only if (reuse value) > (its build cost)" — augmentation: 4h cost, reusable across products → net positive; headline: 40h+8 GPU-days, zero product value → net negative. Logical check holds. ✓

Cell H — The exam-style twist

Worked example

Steps.

Identify the axis. Why the last axis (across keys)? Each query must produce a probability distribution over the keys it looks at — so the numbers in one query's row must sum to 1. Softmaxing down columns would wrongly make each key sum to 1 across queries. This is the classic trap.
Exponentiate the row $[2, 0, - 2]$ : $e^{2} \approx 7.389$ , $e^{0} = 1$ , $e^{- 2} \approx 0.1353$ . Why exponentiate? It turns any real scores into positive numbers whose ratios encode the gaps, and it makes the largest score dominate smoothly.
Sum $= 7.389 + 1 + 0.1353 = 8.524$ . Divide each: $7.389/8.524 \approx 0.867$ , $1/8.524 \approx 0.1173$ , $0.1353/8.524 \approx 0.01587$ . Why divide by the sum? That normalises them to add to exactly 1.

Answer: weights $\approx [0.867, 0.117, 0.0159]$ , which sum to 1 across the row (over keys). The twist tested: softmax runs along the key axis (last dim), per query.

Verify: the three weights sum to $1$ (up to rounding), and the largest raw score gets the largest weight. ✓

Recall Rapid-fire scenario recall

Attention output shape given input $(4, d_{k})$ ? ::: $(4, d_{k})$ — attention rewrites tokens, keeps count and width. Softmax of a single-element row equals? ::: exactly $1$ , for any value. Which degenerate sequence actually crashes attention? ::: the empty (zero-length) one — its softmax is a $0/0$ ; length-1 is safe. The $+ I$ in the residual gradient fails to protect the gradient only when? ::: the learned part's gradient equals exactly $- 1$ , which cancels the identity. Divisor that pins attention-score spread to 1 for all $d_{k}$ ? ::: $d_{k}$ . A 1.74-sigma gap from the paper means? ::: within normal seed noise — not a bug. Softmax over which axis in attention? ::: the key axis (last dim), so each query's row sums to 1.