Before the traps can trap you, you need the handful of objects they lean on. The figures below define each one from scratch — read them once, top to bottom.
Recall What the three key words mean (build these first)
Replication ::: Re-running the exact same code on the exact same data — you check the pipeline runs and lands on the reported number.
Reproduction ::: Rebuilding the method from the written description with your own code; details may differ, so you expect a small gap, not an exact match.
Ablation ::: Deliberately deleting or disabling one claimed component to see whether performance actually drops — the test of why it works, not just that it works.
Every "which pass?" trap below refers to this pipeline. Pass 1 is a 10-minute skim — read only title, abstract, section headings, figures, and conclusion, and decide whether the paper is worth your time. Pass 2 is a ~1-hour deep read — follow the argument, extract the one core equation, but skip the proofs. Pass 3 is the multi-hour reproduction — rebuild the method in code and check it against reported numbers. The passes are a funnel: each keeps fewer papers but spends more time on them.
Several traps (TF8, SE1, WHY1) talk about Q, K, V, dk, and dv. Here is the whole vocabulary on one picture. Think of attention as a soft lookup table: every position asks a query and every position advertises a key; the query dotted with each key gives a match score, and those scores decide how much of each position's value you carry away.
First, one piece of notation that appears in the formula. The little superscript T in KT means transpose — flip the grid of numbers over its diagonal so its rows become columns. We need it purely so the shapes line up for a dot product: it turns "list of keys standing up" into "list of keys lying down" so each query can be multiplied against every key at once.
Q (query), K (key), V (value) are lists of vectors — one per position in the sequence.
dk is the length of each query and key vector (how many numbers it holds). Queries and keys must share this length so their dot product is defined.
dv is the length of each value vector — it need not equal dk. Because the output is a weighted blend of value vectors, the output has length dv: the value dimension, not the key dimension, sets the width of what attention hands back.
KT is Ktransposed (rows become columns) so that QKT is the grid of all query·key dot products (the raw match scores).
softmax turns a row of raw scores into a set of positive weights that sum to 1 — a probability distribution saying "how much attention to pay to each position."
Attention(Q,K,V)=softmax(dkQKT)V
Figure 2 — attention as a soft lookup (Q asks, K answers, V is fetched):
Figure 3 — variance grows with dk; the dk scaling keeps softmax sensitive:
A residual block doesn't ask the layers to produce the whole output H(x). Instead the layers produce only a correctionF(x), and a wire called the identity shortcut carries the input x straight across and adds it back:
H(x)=F(x)+x
Because there is a direct x path, the derivative always contains a +I term (the "identity" I just means "multiply by 1 in every direction"), so gradient flows straight backward even when the learned part's gradient shrinks to zero. A ResNet-18 is a small version with 18 such layers; ResNet-152 stacks 152 — hence "start small when reproducing."
Figure 4 — the residual block and its gradient shortcut:
TF1 — "If a paper is peer-reviewed, its reported numbers are trustworthy enough to skip checking error bars."
False. Peer review checks novelty and soundness of argument, not that every number replicates; ML has a known reproducibility crisis, so you still inspect sample size, seeds, and error bars. See Statistical Significance Testing.
TF2 — "Reproduction and replication are the same thing."
False. Replication reruns the original code/data exactly; reproduction rebuilds the method from the description with your own code, so a small performance gap is expected and acceptable.
TF3 — "A single run beating the baseline by 0.3 points proves the method is better."
False. One run has no variance estimate; the gap could be seed noise. You need multiple seeds and a significance test before claiming improvement.
TF4 — "The first pass of the three-pass method should give you enough to reproduce the paper."
False. Pass 1 is a 10-minute decision pass (is this worth my time?); reproduction is Pass 3 and takes hours — expecting mastery from Pass 1 is the classic "understand everything at once" mistake.
TF5 — "You should read a paper linearly, top to bottom, like a textbook."
False. Papers front-load claims and back-load evidence; jumping to figures, tables, and the core equation first is faster, and proofs can wait until they're the contribution you care about.
TF6 — "If your reproduction doesn't match the paper's number, the paper must be wrong."
False. The far more likely cause is a difference in your pipeline — data preprocessing, initialization, or LR schedule — which the paper often under-specifies. Suspect your own setup before the authors'.
TF7 — "An ablation that removes a component and shows a drop proves that component is necessary."
Mostly true but with a caveat: it shows the component helps in this configuration. Interactions can mean the drop shrinks or vanishes once other pieces change, so ablations are evidence, not proof of universal necessity.
TF8 — "Because QKT has variance dk, the dk scaling in attention is just cosmetic."
False. The scaling restores variance to 1; without it, large dot products push the softmax into a near-one-hot region where gradients vanish, so scaling directly affects trainability. See Attention Mechanisms.
TF9 — "If a model can overfit 10 training samples, the whole training pipeline is correct."
False. It confirms the loss and backprop can drive the loss to zero — a necessary sanity check — but says nothing about your data augmentation, splits, or generalization. It rules out one class of bugs, not all.
TF10 — "Reusing a pretrained backbone from the paper means you no longer need to reproduce anything."
False. Using Transfer Learning weights checks the inference path, but the paper's contribution may be the training procedure; reproducing that still requires the full loop from scratch.
TF11 — "The output of attention always has the same length as the query vectors."
False. The output is a blend of value vectors, so its length is dv, the value dimension; only when dv=dk do they coincide.
SE1 — "The paper says entries are N(0,1), so Var(QKT)=Var(qiki)=1."
Error: the dot product is a sum of dk independent products, so variances add: Var(QKT)=dk. Forgetting to multiply by dk hides exactly the effect the dk scaling corrects.
SE2 — "I couldn't match the baseline, so I skipped that and reproduced the new method's number instead."
Error: the baseline check is the prerequisite verification. If your baseline is off, any gap for the new method is uninterpretable — you can't tell innovation from pipeline bug.
SE3 — "The residual block is H(x)=F(x), and the skip just makes it deeper."
Error: it's H(x)=F(x)+x. The +x (identity shortcut) is the whole point — its gradient contributes a +I term so gradients flow backward undiminished.
SE4 — "To reproduce ResNet-152, I'll start by building the full 152-layer model."
Error: start with the smallest variant (ResNet-18). Debugging shapes and the training loop is far cheaper on a small model before scaling depth.
SE5 — "I logged only the final accuracy, so I can compare runs later."
Error: without per-run seeds, hyperparameters, and config saved, runs aren't comparable and results aren't reproducible. Track configs and seeds — see Experiment Tracking and Versioning.
SE6 — "Masking padding just means setting those attention scores to 0 before the softmax."
Error: 0 is a middling score, so softmax would still hand padding real weight. You add a very large negative number (in code often −109, conceptually −∞; most libraries expose a masked-fill / additive-mask constant) so that after softmax the weight is effectively zero.
SE7 — "My method beats baseline on the test set I tuned on, so I'm done."
Error: tuning on the test set leaks it into training — the number is optimistic. Tune on validation, report on a held-out test set touched once. See Hyperparameter Tuning.
SE8 — "The paper omitted the initialization scheme, so it doesn't matter."
Error: initialization (Xavier/Kaiming) can decide whether a deep net trains at all. An omission means you must infer or sweep it, not ignore it — it's a top hiding spot for reproduction gaps.
Because the standard deviation of the dot product scales as dk (variance is dk, and std =variance); dividing by dk normalizes the spread to 1, whereas dividing by dk would over-shrink it below 1.
WHY2 — Why does the +I term in a residual block prevent vanishing gradients?
Because ∂H/∂x=∂F/∂x+I always includes an identity path, so even if ∂F/∂x→0, gradient still flows straight back through the shortcut.
WHY3 — Why is "can it overfit 10 samples?" a good first reproduction check?
Because a correct loss and backprop should memorize a tiny set perfectly; failure to do so isolates a bug in the loss or gradient path before you waste hours on full training.
WHY4 — Why forecast the solution before reading the methods section?
Because writing your own attempt first turns reading into active checking — you notice which of the authors' choices are surprising, which is where the real insight usually hides.
WHY5 — Why do ablation studies matter more than a single headline number?
Because the headline shows that the system works; the ablation shows which component causes it — without it you can't tell whether the new idea or an unrelated tweak drove the gain.
WHY6 — Why is the data pipeline the most bug-prone part of reproduction?
Because preprocessing, normalization, augmentation, and split choices are often only half-described, silently change the effective task, and produce plausible-but-wrong numbers with no crash to alert you.
WHY7 — Why prefer reproducing on the smallest model variant first?
Because it isolates logic bugs (shapes, loss, loop) at low compute cost; scaling depth only after correctness is verified saves both time and GPU memory.
WHY8 — Why can Adam vs SGD change whether a reproduction matches?
Because optimizer choice alters the effective learning-rate dynamics and implicit regularization; a paper tuned for SGD+momentum may not converge to the same point under Adam, opening a real accuracy gap.
The figure below organizes the boundary cases as a quick-scan table; the reveals underneath give the reasoning.
Figure 5 — edge-case cheat sheet:
EC1 — What does the attention softmax do if all key scores are equal (e.g. a fresh, untrained layer)?
It outputs a uniform distribution over keys, so the output is just the average of all values — attention starts "paying equal attention to everything" and only sharpens as training breaks the symmetry.
EC2 — In a residual block, what happens if the optimal mapping is the identity, H(x)=x?
The network only needs to learn F(x)=0, i.e. drive its weights to zero — far easier than learning a full identity transform, which is exactly why deep residual nets don't degrade.
EC3 — What if dk=1 in scaled dot-product attention?
The scaling factor is 1=1, so scaling does nothing and the variance is already 1; the correction only becomes important as dk grows and dot products spread wider.
EC4 — What if your reproduction exactly matches the paper to many decimals without you using their code?
Be suspicious — a perfect match from an independent rebuild is unlikely given seed and pipeline differences; it usually means you copied a hidden dependency or are unknowingly evaluating on their released outputs.
EC5 — What if an ablation removes a component and performance improves?
Then the claimed component isn't helping in your setup — either the paper's benefit was configuration-specific, or you have a bug elsewhere; it's a red flag worth investigating, not silently ignoring.
EC6 — What if the masked positions are ALL of a row (a fully-padded query)?
Every key score in that row becomes a large negative, so softmax over identical values gives a uniform distribution; the resulting output is a meaningless average of value vectors. Such fully-masked rows must be detected and dropped downstream (or zeroed out), never fed forward as if they carried real information.
EC7 — What if two runs of the "same" experiment differ by more than the paper's error bar?
Then either your variance estimate (too few seeds) or your pipeline is unstable; a spread wider than the reported bar means the claimed improvement may not be statistically significant.
EC8 — What if a paper reports no error bars at all?
Treat the numbers as a single sample: you cannot judge significance, so reproduce with several seeds yourself and report a range before drawing conclusions — see Model Debugging for isolating the source of variance.
The three key words are also three rungs you climb in order. You never ablate a claim you cannot yet reproduce, and you never trust a reproduction whose pipeline you have not first replicated. The figure spells out each rung and the check that lets you step up.
