Compute-data-parameter tradeoffs: Worked examples

This page is a case-completeness drill for the parent note Compute–Data–Parameter Tradeoffs. We already know two engine equations from there:

Before touching a single number, let me define the symbols in plain words, because the whole page leans on them:

If any of these feel shaky, re-read the parent's resource triangle first — this page assumes only these four definitions and nothing else.

The scenario matrix

Every problem this topic throws at you falls into one of these cells. The worked examples below are labelled by cell, and together they hit all of them.

Cell (flag)	What varies / what's degenerate	Example
A. Forward direction	Given $N$ and $D$ , find $C$	Ex 1
B. Backward direction	Given $C$ + the 20× rule, find $N$ and $D$	Ex 2
C. Ratio diagnosis	Given $N, D$ , judge over/under-trained	Ex 3
D. Scaling / limiting behaviour	Multiply $C$ by a factor, predict growth	Ex 4
E. Degenerate input	$D \to \infty$ , $N \to \infty$ , or a term $\to 0$	Ex 5
F. Real-world word problem	GPU-hours, wall-clock, dollars	Ex 6
G. Deployment twist	Inference cost flips the "optimal" answer	Ex 7
H. Exam trap	The "6", FMAs, units, wrong exponent	Ex 8

Figure — Compute-data-parameter tradeoffs

The figure above is the map: the curved blue line is the loss-vs- $N$ trench at fixed compute, and each coloured flag is one cell of the matrix. As you work each example below I will point back to its flag — the yellow flag is the optimum (cell B), the red flag is the data-starved big model (cell C, ratio $r < 20$ ), the green flag is the small over-trained model (cell G), and the white dashed floor is the degenerate limit (cell E). Keep glancing back at it.

A — Forward direction: given $N$ and $D$ , find $C$

Worked example Example 1 (cell A) — cost of one training run

A model has $N = 1.3 \times 1 0^{9}$ parameters and reads $D = 2.6 \times 1 0^{10}$ tokens. What compute $C$ does training cost? On the map this run sits at the yellow flag — it will turn out exactly optimal.

Forecast: guess the order of magnitude before reading on — is it closer to $1 0^{18}$ or $1 0^{22}$ FLOPs?

Step 1 — pick the engine. Why this step? We are handed both $N$ and $D$ and asked for $C$ ; that is exactly the forward use of $C \approx 6 N D$ . $C \approx 6 \times (1.3 \times 1 0^{9}) \times (2.6 \times 1 0^{10})$

Step 2 — multiply the mantissas and add the exponents. Why? Scientific notation makes big multiplications safe: multiply $6 \times 1.3 \times 2.6$ , then combine $1 0^{9} \cdot 1 0^{10} = 1 0^{19}$ . $6 \times 1.3 \times 2.6 = 20.28, C \approx 20.28 \times 1 0^{19} \approx 2.0 \times 1 0^{20} FLOPs$

Verify: ratio check — $D / N = 2.6 \times 1 0^{10} /1.3 \times 1 0^{9} = 20$ . That is exactly Chinchilla-optimal, so this is a sensible run (it lands on the yellow flag), and $2 \times 1 0^{20}$ FLOPs sits in the "small research model" range. Units: (dimensionless $N$ )×(tokens)×(FLOPs/param/token) = FLOPs. ✓

B — Backward direction: given $C$ , find $N$ and $D$

Worked example Example 2 (cell B) — how to spend a budget

You are granted $C = 1.5 \times 1 0^{21}$ FLOPs. Using the compute-optimal rule $D \approx 20 N$ , what $N$ and $D$ should you choose? This is the yellow flag on the map — the bottom of the loss trench.

Forecast: will $N$ be nearer 1 billion or 100 billion? Guess now.

Step 1 — collapse two unknowns into one. Why? $C \approx 6 N D$ has two unknowns; the rule $D = 20 N$ removes one, leaving a single-variable equation we can actually solve. $C \approx 6 N (20 N) = 120 N^{2}$

Step 2 — solve for $N$ . Why square-root? Because $C$ is proportional to $N^{2}$ ; undoing a square is a square root. $N \approx \frac{C}{120} = \frac{1.5 \times 1 0 ^{21}}{120} = 1.25 \times 1 0^{19} \approx 3.54 \times 1 0^{9}$

Step 3 — recover $D$ . Why? The rule ties them: $D = 20 N$ . $D = 20 \times 3.54 \times 1 0^{9} \approx 7.07 \times 1 0^{10}$

Verify: plug back — $6 N D \approx 6 \times 3.54 \times 1 0^{9} \times 7.07 \times 1 0^{10} \approx 1.50 \times 1 0^{21}$ ✓ recovers the budget. So ≈3.5B params, ≈71B tokens.

See Chinchilla vs GPT-3 for why this 20:1 split beats the alternatives.

C — Ratio diagnosis: is a model over- or under-trained?

Worked example Example 3 (cell C) — three models, one glance

Diagnose each: (i) $N = 175 B$ , $D = 300 B$ ; (ii) $N = 7 B$ , $D = 1 T$ ; (iii) $N = 70 B$ , $D = 1.4 T$ . On the map, (i) sits at the red flag (data-starved) and (ii) at the green flag (over-trained).

Forecast: which of the three is "just right"?

Step 1 — compute tokens-per-parameter $r = D / N$ for each. Why this metric? Loss depends on the balance $D / N$ , and Chinchilla says the balance point is $r \approx 20$ . The ratio is the one number that tells over- from under-training. $r_{i} = \frac{300}{175} \approx 1.7, r_{ii} = \frac{1000}{7} \approx 143, r_{iii} = \frac{1400}{70} = 20$

Step 2 — read the verdict against $r = 20$ . Why? $r < 20$ means too big for its data (data-starved, the $B / D^{β}$ term dominates); $r > 20$ means over-trained (great for cheap inference); $r = 20$ is training-compute-optimal.

(i) $r \approx 1.7 ≪ 20$ → over-parameterized / data-starved (this is GPT-3, the red flag).
(ii) $r \approx 143 ≫ 20$ → deliberately over-trained (small model, cheap to serve — the LLaMA strategy, the green flag).
(iii) $r = 20$ → Chinchilla-optimal (the yellow flag).

Verify: the parent note states GPT-3 was "~12× too few tokens". Check: $20/1.7 \approx 11.8 \approx 12$ ✓. See Overfitting and capacity for why a starved big model wastes capacity.

D — Scaling / limiting behaviour

Worked example Example 4 (cell D) — you get 10× the compute

Your compute jumps by a factor of $10$ . By what factor should $N$ grow, and by what factor should $D$ grow? Geometrically, the yellow flag slides down and to the right along the trench as compute grows.

Forecast: does the model grow more or less than the data?

Step 1 — use the power laws, not the raw sizes. Why? We only care about the ratio of new to old, and $N_{opt} \propto C^{a}$ means $N_{new} / N_{old} = (C_{new} / C_{old})^{a}$ — the constants cancel. $factor_{N} = 1 0^{a} = 1 0^{0.46}, factor_{D} = 1 0^{b} = 1 0^{0.54}$

Step 2 — evaluate. Why these numbers? $a \approx 0.46, b \approx 0.54$ from the parent's fit. $1 0^{0.46} \approx 2.88, 1 0^{0.54} \approx 3.47$ Model grows ≈2.9×, data grows ≈3.5×. Data grows slightly more because $b > a$ .

Verify (the a+b=1 sanity check): the two growth factors must multiply to the compute factor, since $C \propto N D$ : $2.88 \times 3.47 \approx 10.0 = compute factor ✓$ This is the invariant to always test — if the product isn't the compute factor, you mis-typed an exponent. See Neural Scaling Laws.

E — Degenerate / limiting inputs

Before we can push things to infinity, we must know what every symbol in the loss law means — otherwise " $A / N^{α} \to 0$ " is a magic trick, not maths.

Definition The Chinchilla loss law, symbol by symbol

$L (N, D) = floor E + too-small penalty \frac{A}{N ^{α}} + too-little-data penalty \frac{B}{D ^{β}}$ This is an empirical fit from Hoffmann et al. (2022) — they trained hundreds of models and found this shape matches the data. Each symbol:

$E$ = irreducible loss, the entropy of language you can never beat (a positive constant). Fitted $E \approx 1.69$ nats.
$A$ = size of the parameter penalty — a positive scale constant multiplying the $1/ N^{α}$ term. Fitted $A \approx 406$ .
$α$ = how fast the parameter penalty shrinks as $N$ grows (the exponent on $N$ ). Fitted $α \approx 0.34$ . Bigger $α$ ⇒ extra parameters help more.
$B$ = size of the data penalty — a positive scale constant on the $1/ D^{β}$ term. Fitted $B \approx 410$ .
$β$ = how fast the data penalty shrinks as $D$ grows (the exponent on $D$ ). Fitted $β \approx 0.28$ . Both $α, β$ are strictly between $0$ and $1$ , which is why returns diminish: doubling $N$ does not halve its penalty.

Worked example Example 5 (cell E) — what happens at the extremes?

Using the loss law above, evaluate the four degenerate cases. On the map, all of these press the trench down toward the white dashed floor $E$ .

Forecast: can loss ever reach $0$ ?

Step 1 — send $N \to \infty$ at fixed $D$ . Why? This isolates whether "infinite model" alone wins. The term $A / N^{α} \to 0$ because a positive constant $A$ over an ever-growing $N^{α}$ (positive power) shrinks to nothing. $L \to E + 0 + \frac{B}{D ^{β}}$ Interpretation: an infinitely large model still carries a data penalty $B / D^{β}$ . Size alone cannot save you.

Step 2 — send $D \to \infty$ at fixed $N$ . Why? The mirror case. Now $B / D^{β} \to 0$ . $L \to E + \frac{A}{N ^{α}} + 0$ Infinite data still leaves the parameter penalty. This is the capacity ceiling — the flaw behind "just train on more data forever".

Step 3 — send both $\to \infty$ . Why? The joint limit. $L \to E$ You bottom out at the irreducible loss $E$ : the genuine unpredictability of language. Loss never reaches $0$ .

Step 4 — degenerate zero input. Why cover it? Exams love $D = 0$ . With $D = 0$ , $B / D^{β} \to \infty$ (division by zero) — a model that has read nothing has unbounded loss. Likewise $N = 0$ (no parameters) gives $A / N^{α} \to \infty$ . Both are formally infinite, matching intuition: nothing in, no learning.

Verify: with $E = 1.69, A = 406, α = 0.34, B = 410, β = 0.28$ , take a big-but-finite $N = 1 0^{12}$ and $D = 1 0^{13}$ : $N^{α} = (1 0^{12})^{0.34} \approx 1.2 \times 1 0^{4}$ so $A / N^{α} \approx 406/1.2 \times 1 0^{4} \approx 0.034$ ; $D^{β} = (1 0^{13})^{0.28} \approx 4.4 \times 1 0^{3}$ so $B / D^{β} \approx 410/4.4 \times 1 0^{3} \approx 0.094$ . Thus $L \approx 1.69 + 0.034 + 0.094 \approx 1.82$ , still hugging $E = 1.69$ from above ✓.

F — Real-world word problem

Worked example Example 6 (cell F) — GPUs, wall-clock, and dollars

You rent 1024 GPUs. Each delivers $4 \times 1 0^{14}$ FLOP/s of usable throughput (after a 40% efficiency haircut, this is the real number). You want the $C = 1.5 \times 1 0^{21}$ FLOP run from Example 2 (still the yellow flag). How many days does it take, and at $2/GPU-hour, what does it cost?

Forecast: hours, days, or weeks?

Step 1 — total throughput of the cluster. Why? Time = work ÷ rate, and the cluster's rate is per-GPU rate times number of GPUs. $R = 1024 \times 4 \times 1 0^{14} = 4.096 \times 1 0^{17} FLOP/s$

Step 2 — wall-clock seconds, then hours, then days. Why divide? Total FLOPs $C$ divided by FLOPs-per-second gives seconds; divide by $3600$ s/h for hours, then by $24$ h/day for days. $t = \frac{1.5 \times 1 0 ^{21}}{4.096 \times 1 0 ^{17}} \approx 3662 s = \frac{3662}{3600} \approx 1.017 h = \frac{1.017}{24} \approx 0.042 days$ So the run finishes in about 1 hour, i.e. ≈0.042 days — a fraction of a single day.

Step 3 — GPU-hours and dollars. Why GPU-hours? Billing is per GPU per hour: $1024$ GPUs $\times 1.017$ h. $GPU-h = 1024 \times 1.017 \approx 1041, cost = 1041 \times $2 \approx $2082$

Verify: cross-check via GPU-hours × per-GPU rate = total FLOPs. $1041 GPU-h \times 3600 s/h \times 4 \times 1 0^{14} FLOP/s \approx 1.5 \times 1 0^{21}$ ✓. So ≈0.042 days (≈1 hour), ≈$2.1k.

G — Deployment twist: inference flips the answer

Worked example Example 7 (cell G) — training-optimal ≠ serving-optimal

Two candidates serve the same task: Model X ( $N_{X} = 70 B$ , Chinchilla-optimal, the yellow flag) and Model Y ( $N_{Y} = 7 B$ , over-trained on 10× more data, the green flag). Each inference token costs $\approx 2 N$ FLOPs (forward pass only). If you expect to serve $I = 5 \times 1 0^{14}$ tokens over the model's life, which wins on total (train + serve) compute?

Forecast: does the smaller model's cheaper inference beat its extra training?

Step 1 — inference cost of each. Why $2 N$ , not $6 N$ ? Serving is forward-pass only — no backward pass, no gradients — so it is $2 N$ FLOPs per token, one-third of training's $6 N$ . $C_{X}^{inf} = 2 N_{X} I = 2 \times 7 \times 1 0^{10} \times 5 \times 1 0^{14} = 7.0 \times 1 0^{25}$ $C_{Y}^{inf} = 2 N_{Y} I = 2 \times 7 \times 1 0^{9} \times 5 \times 1 0^{14} = 7.0 \times 1 0^{24}$ (I wrote $N_{X} = 70 B = 7 \times 1 0^{10}$ , $N_{Y} = 7 B = 7 \times 1 0^{9}$ .)

Step 2 — the smaller model is 10× cheaper per served token. Why? Inference cost scales linearly with $N$ ; $N_{Y}$ is $10 \times$ smaller, so serving is $10 \times$ cheaper. Over $5 \times 1 0^{14}$ tokens that saved compute is $6.3 \times 1 0^{25}$ FLOPs — far larger than any realistic extra training cost (LLaMA-7B trained on ~1T tokens is only $6 \times 7 \times 1 0^{9} \times 1 0^{12} \approx 4 \times 1 0^{22}$ FLOPs).

Step 3 — conclusion. Why does this contradict Chinchilla? Chinchilla minimizes training loss per training FLOP. When inference dominates the lifetime budget, you deliberately pick a smaller, over-trained model — the parent's third mistake made concrete.

Verify: inference saving $= C_{X}^{inf} - C_{Y}^{inf} = 7.0 \times 1 0^{25} - 7.0 \times 1 0^{24} = 6.3 \times 1 0^{25}$ FLOPs, which dwarfs the ≈ $4 \times 1 0^{22}$ FLOP extra training — Model Y wins by ~1000×. ✓ See Inference cost vs training cost.

H — Exam trap: the "6", FMAs, and units

Worked example Example 8 (cell H) — spot the four traps

An exam claims: "A run does $C = 1.2 \times 1 0^{21}$ FMAs. Since $C \approx 6 N D$ and $D = 20 N$ , solve for $N$ ." A student writes $N = C /120 = 1 0^{19} \approx 3.16 \times 1 0^{9}$ . What went wrong?

Forecast: is the student's $N$ too big or too small?

Step 1 — Trap 1, units mismatch: FMA vs FLOP. Why fatal? The "6" in $6 N D$ counts FLOPs, but the problem gave FMAs. One FMA = 2 FLOPs, so $C_{FLOP} = 2 \times 1.2 \times 1 0^{21} = 2.4 \times 1 0^{21}$ FLOPs. Using the raw FMA number under-counts compute by 2×.

Step 2 — redo with correct units. Why? Now the "6" and the compute are both in FLOPs. $N = \frac{C _{FLOP}}{120} = \frac{2.4 \times 1 0 ^{21}}{120} = 2.0 \times 1 0^{19} \approx 4.47 \times 1 0^{9}$ The correct $N \approx 4.47 B$ — the student's $3.16 B$ was too small.

Step 3 — name the other three traps. Why list them? Exams stack them; all four must be spotted.

Trap 2 — the "6" is leading-order only. It ignores attention FLOPs (which grow with sequence length), embeddings, and layernorms. Never call $6 N D$ exact — this is why we write " $\approx$ ", not " $=$ ".
Trap 3 — mixing loss exponents with allocation exponents. Do not swap $α, β$ (exponents in the loss law) with $a, b$ (exponents in the optimal allocation). Here $a = β / (α + β) \approx 0.46$ , not $α$ itself.
Trap 4 — the $D = 20 N$ rule is training-only. It minimizes training loss per FLOP; if inference dominates (cell G), the optimal ratio shifts well above 20. Applying "20" blindly to a heavily-served model is wrong.

Conclusion: the student's $N = 3.16 \times 1 0^{9}$ is wrong; the correct answer is $N \approx 4.47 \times 1 0^{9}$ (≈4.47B, about 41% larger). The single decisive error was the FMA→FLOP units slip in Trap 1 — the ratio of correct to wrong answer is exactly $4.47/3.16 = 2 \approx 1.41$ , the fingerprint of a missing factor of 2 under a square root.

Verify: ratio of the two answers $= 4.47/3.16 = 1.414 = 2$ — exactly the FMA-to-FLOP factor, confirming the single arithmetic error was the units. ✓ See FLOP accounting in Transformers.

Recall Rapid self-test (cover the answers)

Given $N = 1.3 B$ , $D = 26 B$ , what is $C$ ? ::: $6 N D \approx 2.0 \times 1 0^{20}$ FLOPs. Budget $C = 1.5 \times 1 0^{21}$ , rule $D = 20 N$ : find $N$ . ::: $C /120 \approx 3.5 \times 1 0^{9}$ . $D / N = 1.7$ — over- or under-trained? ::: Over-parameterized / data-starved (like GPT-3). 10× compute: model growth factor? ::: $1 0^{0.46} \approx 2.9 \times$ (data $1 0^{0.54} \approx 3.5 \times$ ). As $N \to \infty$ and $D \to \infty$ , loss tends to? ::: The irreducible loss $E$ (never $0$ ). Serving cost per token in FLOPs? ::: $\approx 2 N$ (forward pass only), one-third of training's $6 N$ . $C$ given as FMAs — convert before using $6 N D$ ? ::: Yes, multiply by 2 (1 FMA = 2 FLOPs).