Neural scaling laws (Chinchilla, compute-optimal): Worked examples

The scenario matrix

Every problem in this topic is one of these cells. The examples below are labelled by cell.

#	Cell (scenario class)	What makes it distinct	Example
C1	Balanced optimum	$N, D$ chosen at the compute-optimal split	Ex. 1
C2	Data-starved ( $D / N ≪ 20$ )	Too big a model, too little data — $B / D^{β}$ dominates	Ex. 2
C3	Data-glutted ( $D / N ≫ 20$ )	Too small a model, too much data — $A / N^{α}$ dominates	Ex. 3
C4	Scale-up map	Given $k \times$ compute, how to re-split $N, D$	Ex. 4
C5	Forecast a loss drop	Predict new $L$ before/after scaling	Ex. 5
C6	Limiting / degenerate	$D \to \infty$ or $N \to \infty$ : what does $L$ approach?	Ex. 6
C7	Diminishing-returns arithmetic	"How much compute to halve the reducible loss?"	Ex. 7
C8	Real-world word problem	Dollars, GPU-hours, a deadline	Ex. 8
C9	Exam twist	Sum-of-exponents trap / the "forget the 6" trap	Ex. 9

Prerequisite links if a cell feels shaky: Compute budgets & FLOPs, Overfitting vs underfitting, Kaplan scaling laws.

Figure — Neural scaling laws (Chinchilla, compute-optimal)

The figure above is our map: the black curves are lines of equal compute (each point on one curve has the same $C = 6 N D$ ). The red curve is the compute-optimal ridge — the best $(N, D)$ for each budget. Cells C2 and C3 are the two ways to fall off that red ridge.

Worked examples

Worked example Ex. 1 — Balanced optimum (cell C1)

You have $C = 6 \times 1 0^{23}$ FLOPs. Using the heuristic $D \approx 20 N$ , find $N$ and $D$ .

Forecast: guess — will $N$ be closer to $10$ B or $100$ B? (Feel it before computing.)

Write both constraints. $C = 6 N D$ and $D = 20 N$ . Why this step? Two equations, two unknowns — a solvable system. The second is the Chinchilla rule of thumb.
Substitute $D = 20 N$ into $C = 6 N D$ . $C = 6 N (20 N) = 120 N^{2}$ . Why? Eliminate $D$ so we have one unknown $N$ .
Solve for $N$ . $N = C /120 = 6 \times 1 0^{23} /120 = 5 \times 1 0^{21} \approx 7.07 \times 1 0^{10}$ . Why? Just algebra on step 2. So $N \approx 71$ B params.
Back out $D$ . $D = 20 N \approx 1.41 \times 1 0^{12} = 1.4$ T tokens. Why? Plug $N$ into $D = 20 N$ .

Verify: $6 N D = 6 (7.07 \times 1 0^{10}) (1.41 \times 1 0^{12}) \approx 6.0 \times 1 0^{23}$ ✓ matches the budget. Notice this reproduces the real Chinchilla model ( $\approx 70$ B, $1.4$ T). The tokens-per-param ratio is $D / N = 20$ by construction — dead centre of the red ridge.

Worked example Ex. 2 — Data-starved (cell C2)

Gopher-style: $N = 280$ B, $D = 300$ B tokens. Is it under- or over-sized? By how much is data short?

Forecast: is $D / N$ above or below $20$ ?

Compute the ratio. $D / N = 300 B /280 B \approx 1.07$ tokens/param. Why? The ratio vs the target $20$ instantly tells you which term dominates.
Compare to the ideal. $1.07 ≪ 20$ → severely data-starved; the $B / D^{β}$ penalty is large. Why? Small $D$ ⇒ big $B / D^{β}$ . This is the over-model-under-data corner (Overfitting vs underfitting intuition: it's not classic overfit but wasted capacity).
What is its compute? $C = 6 N D = 6 (2.8 \times 1 0^{11}) (3 \times 1 0^{11}) = 5.04 \times 1 0^{23}$ FLOPs. Why? We need $C$ to find the balanced alternative.
Rebalance at the same $C$ using Ex. 1's method: $N^{⋆} = C /120 = 5.04 \times 1 0^{23} /120 \approx 6.48 \times 1 0^{10} \approx 65$ B, and $D^{⋆} = 20 N^{⋆} \approx 1.30 \times 1 0^{12}$ . Why? Shows the fix: shrink $N$ by ~4×, grow $D$ by ~4×, at nearly the same cost.

Verify: $6 N^{⋆} D^{⋆} = 6 (6.48 \times 1 0^{10}) (1.30 \times 1 0^{12}) \approx 5.05 \times 1 0^{23}$ ✓ same budget. A 4× smaller, better-fed model — exactly the Chinchilla story.

Worked example Ex. 3 — Data-glutted (cell C3)

A tiny model $N = 1$ B trained on $D = 500$ B tokens. Which term dominates the loss? Is this efficient?

Forecast: $D / N = ?$ — will it be far above or below $20$ ?

Ratio. $D / N = 500 B /1 B = 500$ tokens/param. Why? Compare to $20$ : this is $25 \times$ too data-heavy.
Which penalty dominates? $500 ≫ 20$ ⇒ the model is too small ⇒ $A / N^{α}$ dominates; extra data barely helps. Why? Big data drives $B / D^{β}$ tiny, but the brain can't represent more — capacity bottleneck.
Compute wasted? $C = 6 (1 0^{9}) (5 \times 1 0^{11}) = 3 \times 1 0^{21}$ FLOPs. Optimal $N$ for this $C$ : $3 \times 1 0^{21} /120 \approx 5.0 \times 1 0^{9} = 5$ B, i.e. you should have used a 5× bigger model. Why? Confirms inefficiency: same compute, better split available.

Verify: balanced $D^{⋆} = 20 \times 5 \times 1 0^{9} = 1 0^{11} = 100$ B, and $6 (5 \times 1 0^{9}) (1 0^{11}) = 3 \times 1 0^{21}$ ✓ same budget. Lesson: both starvation (C2) and glut (C3) are off the red ridge.

Worked example Ex. 8 — Real-world word problem (cell C8)

Your cluster does $2 \times 1 0^{18}$ FLOPs/second. You may run for 10 days. Assuming full utilisation, what is the compute-optimal $(N, D)$ , and how many tokens per parameter?

Forecast: roughly what size model — millions, billions, or trillions of params?

Total compute. Seconds in 10 days $= 10 \times 86400 = 8.64 \times 1 0^{5}$ s. $C = 2 \times 1 0^{18} \times 8.64 \times 1 0^{5} = 1.728 \times 1 0^{24}$ FLOPs. Why? Compute $=$ rate $\times$ time; converts a deadline into a budget.
Optimal $N$ (Ex. 1 method): $N = C /120 = 1.728 \times 1 0^{24} /120 = 1.44 \times 1 0^{22} = 1.2 \times 1 0^{11} = 120$ B. Why? Same balanced-split algebra with $D = 20 N$ .
Optimal $D$ . $D = 20 N = 2.4 \times 1 0^{12} = 2.4$ T tokens. Why? Chinchilla rule.
Tokens/param. $D / N = 20$ by construction. Why? Sanity: we built it on the ridge.

Verify: $6 N D = 6 (1.2 \times 1 0^{11}) (2.4 \times 1 0^{12}) = 1.728 \times 1 0^{24}$ ✓ exactly the budget. Units check: $[FLOPs/s] \times [s] = [FLOPs]$ ✓.

Worked example Ex. 9 — Exam twist: the two classic traps (cell C9)

(a) A student writes $N \propto C^{0.45}$ and $D \propto C^{0.60}$ . Spot the error. (b) Another writes $C = N D$ and gets compute $8 \times$ too small. Why?

Forecast: which single check kills (a) instantly?

(a) Sum test. Exponents must satisfy $0.45 + (D-exp) = 1$ because $C = 6 N D$ forces $N \propto C^{a}$ , $D \propto C^{b}$ with $a + b = 1$ . Why? Product $N D \propto C^{a + b}$ must be $\propto C^{1}$ .
Apply. $0.45 + 0.60 = 1.05 \neq = 1$ → impossible. Correct value is $0.55$ . Why? Fails the sum test; the D-exponent was mis-stated.
(b) The missing 6. $C = 6 N D$ , not $N D$ . Dropping the $6$ undercounts by exactly the factor $6$ (2 for multiply-add, ×3 for forward+backward). Why? The $6$ encodes the arithmetic per token per parameter (Compute budgets & FLOPs).
Correct the claim. "8× too small" is loose — the true factor is $6$ , so the student's error is a $6 \times$ undercount. Why? Restore the constant.

Verify: (a) $0.45 + 0.55 = 1.00$ ✓ (the fixed pair sums to 1; the student's $1.05$ did not). (b) ratio $6 N D / (N D) = 6$ ✓.

Recall Self-quiz (reveal after answering)

Compute-optimal $N, D$ for $C = 6 \times 1 0^{23}$ ? ::: $N \approx 71$ B, $D \approx 1.4$ T (Ex. 1). Multiplier on $N$ and $D$ when compute $\times 8$ ? ::: $8^{0.45} \approx 2.55$ and $8^{0.55} \approx 3.14$ (Ex. 4). Doubling balanced $N, D$ drops reducible loss by roughly? ::: ~19–20% (Ex. 5). As $D \to \infty$ with $N$ finite, $L \to$ ? ::: $E + A / N^{α}$ — data penalty gone, floor and param penalty remain (Ex. 6). Factor on $N$ needed to halve the param penalty? ::: $2^{1/0.34} \approx 7.66$ (Ex. 7). Why can't $N$ -exp and $D$ -exp both be 0.6? ::: They must sum to 1 (Ex. 9a).

Related: Kaplan scaling laws (the earlier, over-param-tilted fit), Mixture-of-Experts (changes the effective $N$ vs FLOPs relationship), Learning rate schedules (affects the fitted constants but not the exponent logic).