Foundations — Reading and reproducing ML papers

Every ML paper is written in a compressed language. The parent note assumes you already read that language fluently. Here we slow all the way down and earn each symbol one at a time, in the order they depend on each other — nothing appears in a formula before it has its own section below. (We will only name the two big target equations at the very end, once every piece inside them exists.)

Where the parent topic note lives is the destination; this page is the runway.

1. A number, a list of numbers, and bold letters

Before any equation, know what the pieces are.

• A scalar is a single number, like $3$ or $-0.5$. We write scalars in plain italics: $x$, $d_k$.
• A vector is an ordered list of numbers — think of it as an arrow, or as a column of stacked values. We write vectors in bold: $\mathbf{x}$.

Why the topic needs it. In the parent note, a bold input like $\mathbf{x}$ is what flows into a neural network layer — an image flattened into numbers, a word turned into a list. You cannot read a layer equation without knowing a bold letter is a list of numbers.

2. A matrix: a grid of stacked vectors

We often need many vectors at once. Stack them and you get a matrix.

The picture: rule out a table with $r$ rows and $c$ columns and drop a number in every cell. A single vector of length $n$ is just a matrix of shape ($1$, $n$) or ($n$, $1$).

Why the topic needs it. In the parent note $Q$, $K$, $V$ are matrices — each row is one query, key, or value vector. To combine them we need operations that respect their shapes, which is exactly what the next two sections build.

3. A function: a machine that transforms

The picture: a box with an arrow in and an arrow out. Same input always gives the same output.

• $\mathcal{H}(\mathbf{x})$ = "the whole transformation a layer is supposed to perform" (H for the desired mapping).
• $\mathcal{F}(\mathbf{x})$ = "the residual — the small correction the layer actually learns."

Why the topic needs it. Papers bury their key machine in prose. Recognising "this script letter is a function from input to output" is the single move that lets you extract the core equation in Pass 2.

4. Superscript $T$: flipping a table on its side (transpose)

Now that we have matrices (§2), some sit as rows and some as columns, and to line them up we flip one.

The picture: take a rectangular table of numbers and tip it 90°.

Why the topic needs it. In attention, $Q{K}^T$ appears. $Q$ and $K$ are stacks of vectors (matrices). Transposing $K$ makes the shapes line up so that multiplying them measures how similar each query is to each key — see §5. Without transpose the shapes clash and the multiplication is undefined.

5. The dot product: how much two arrows agree

To turn two vectors into a single similarity number, we use the dot product. (Multiplying $Q$ by $K^T$ is really a table full of these dot products, one per row-pair.)

Why the topic needs it. Multiplying $Q$ by $K^T$ produces a whole matrix of these dot products — one similarity number for every query–key pair. That matrix is exactly what softmax (§6) will turn into attention weights.

6. Softmax: turning a whole list of scores into a choice

We now have raw similarity scores. But scores can be any size, positive or negative. To use them as weights ("how much attention to pay"), we need them to be positive and to sum to $1$ — a probability distribution.

Why the topic needs it. In the parent's attention formula, softmax (applied over the key axis) converts the similarity matrix into attention weights that then blend the value vectors $V$. It is the step that decides what the model looks at.

7. The derivative, the gradient, and the Jacobian: which way is downhill

Training a model = adjusting numbers to make a loss (an error score) as small as possible. To know which way to nudge each number, we need the derivative.

8. Variance and the normal distribution: why divide by $\sqrt{d_k}$

The last symbols in the parent note come from probability.

The picture: a symmetric bell centred at $0$; variance is how wide the bell is.

Why the topic needs it. This single scaling factor is the whole "from first principles" argument in the parent's Transformer example. Note that $\text{Var}$ here is always the variance of one score entry, never of a whole matrix.

Now the two target equations read as sentences

With every piece earned, the parent note's headline formulas decode cleanly:

• ResNet: $\mathcal{H}(\mathbf{x})=\mathcal{F}(\mathbf{x})+\mathbf{x}$ — "the desired transformation $\mathcal{H}$ (a function, §3) of the input vector $\mathbf{x}$ (§1) equals the learned residual $\mathcal{F}$ plus the input added straight back."
• Attention: $\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$ — "matrix-multiply queries $Q$ by keys transposed $K^T$ (§2, §4) to get a table of dot-product similarities (§5); divide each entry by $\sqrt{d_k}$ to keep its variance at $1$ (§8); softmax along the key axis (§6) into attention weights; use them to blend the value rows $V$."

Prerequisite map

Each foundation on the left feeds the reproduction skills on the right. Miss one box and a paper's core equation stays opaque.

Related vault topics

Once these symbols are solid, the parent note connects outward to: Attention Mechanisms (where softmax and $\sqrt{d_k}$ live for real), Transfer Learning, Hyperparameter Tuning and Experiment Tracking and Versioning (the Pass-3 reproduction machinery), Statistical Significance Testing (checking those "within 0.5 BLEU" claims honestly), Model Debugging (the overfit-10-samples sanity check), and finally writing your own results up in Paper Writing and Publication.

Equipment checklist

Test yourself — reveal only after you have answered.