Foundations — Compute-data-parameter tradeoffs

This is the ground-floor page for Compute-data-parameter tradeoffs. The parent note throws around $N$, $D$, $C$, FLOPs, loss $L$, exponents $\alpha ,\beta$, and the "6" in $6ND$ as if you already know them. Here we build every one from nothing, in an order where each rests on the previous. A smart 12-year-old should finish this page ready to read the parent.

0. The absolute starting point: what a model is

Look at the figure: the box has three dials on it. Real models have billions of dials, but the idea is identical. Training = automatically turning the dials until the guesses get good. That's the entire game. Every symbol below is about either the dials, the practice questions, or the effort of turning them.

1. $N$ — the number of parameters

The picture: $N$ is the size of the box in the figure above — a bigger box holds more dials. Why does the topic need $N$? Because more knobs means more capacity (see §5), but every knob also costs effort to train (see §3). $N$ is one of the two things we are trading off.

2. $D$ — the number of tokens

The picture: $D$ is the length of the tape of practice questions feeding into the box. Why does the topic need $D$? Because a model learns by being quizzed on data. More tokens ($D$ larger) = more practice = better learning — up to a point. $D$ is the other thing we trade off against $N$.

3. FLOP — one unit of computer effort

Why this unit and not "seconds"? Seconds depend on how fast your GPU is; FLOPs measure the work itself, independent of hardware. A faster chip does the same FLOPs in less time. So FLOPs let us compare training runs fairly. We need this before we can define the budget $C$.

4. $C$ — the compute budget, and why $C\approx 6ND$

Now the key link. Each token, as it passes through the model, touches every knob a fixed number of times:

Look at the figure: effort $C$ is the area of a rectangle whose sides are $6N$ and $D$. This is the single most important picture in the whole topic — because if the area (budget) is fixed, then making one side longer forces the other side shorter. That's the trade-off, drawn as a rectangle. Every time the parent says "at fixed compute", picture holding this area constant while sliding the sides.

Related deep dives: FLOP accounting in Transformers pins down where the "6" is approximate; Inference cost vs training cost adds a second budget for serving the model.

5. $L$ — the loss, and capacity

Why the topic needs $L$: it's the thing we are trying to make small. $N$ and $D$ are the dials we control; $L$ is the scoreboard. The parent's whole optimisation is "make $L$ as small as possible for a fixed $C$." See Overfitting and capacity for the $N$-vs-$D$ balance, and Neural Scaling Laws for how $L$ falls as you scale.

6. Exponents and $\propto$ — reading power-law shorthand

Look at the figure: three curves, $C^1$ (straight line), $C^{0.5}$ (bends down), $C^{0.54}$. Why does the topic need fractional exponents? Because when you double compute, the model should not double in size — it grows only by $2^{0.46}\approx 1.38$. The fraction is the answer to "how much bigger?" and it's always less than the full compute factor because you split growth between $N$ and $D$.

The prerequisite map

Each box on the left is defined on this page; they all flow into the parent topic at the bottom.

Equipment checklist

Test yourself — you should be able to answer each before opening the parent note.