Neural scaling laws (Chinchilla, compute-optimal): Visual walkthrough

Parent: Neural scaling laws (Chinchilla, compute-optimal)

This page rebuilds the central Chinchilla result — given a fixed pile of compute, how big should the model be and how much data should it eat? — from the ground up, one picture per step. We assume you know nothing except "computers do arithmetic". Every symbol is earned before it is used.

Step 0 — The three quantities we will juggle

Before any math, meet the players. Everything on this page is a story about three numbers.

If you have never seen "FLOP", it is literally one arithmetic step — one multiply, or one add. A modern training run does about $1 0^{23}$ of them. That giant number is $C$ . See Compute budgets & FLOPs for where FLOPs come from.

Figure — Neural scaling laws (Chinchilla, compute-optimal)

The picture shows the three as sliders. The whole derivation is: $C$ is fixed, so if you push $N$ up you must pull $D$ down. We want the sweet spot.

Step 1 — WHAT the loss actually looks like

WHAT: We first need a formula for how wrong the model is. Call it $L$ , the loss — a single number, smaller = better. The parent note gives the empirical shape:

L (N, D) = floor you can’t beat E + too few knobs \frac{A}{N ^{α}} + too little data \frac{B}{D ^{β}}

Reading it term by term, right where each symbol sits:

$E$ — a fixed floor (the irreducible loss). Even a perfect infinite model hits it, because language has genuine randomness. It never changes, so remember it and set it aside.
$\frac{A}{N ^{α}}$ — the penalty for having too few parameters. $A$ is just a size constant. As $N$ (knobs) grows, $N^{α}$ grows, so the fraction shrinks. $α \approx 0.34$ controls how fast.
$\frac{B}{D ^{β}}$ — the penalty for having too little data. Same shape: as $D$ (tokens) grows, this shrinks. $β \approx 0.28$ .

WHY this shape and not, say, $e^{- N}$ ? Because measurements show diminishing returns: each doubling of $N$ removes the same fraction of the reducible error, not a fixed amount. "Constant fractional shrink per doubling" is the exact fingerprint of a power law $1/ N^{α}$ , never an exponential. (An exponential would keep removing a fixed slice forever and hit zero — reality doesn't.)

PICTURE:

Watch the blue curve $A / N^{α}$ fall steeply then flatten — that flattening is diminishing returns. The dashed line is the floor $E$ ; the curve approaches it but never touches.

Step 2 — WHAT ties $N$ and $D$ together: the compute budget

WHAT: Training cost. The parent note states the standard estimate for a dense transformer:

C \approx 6 N D (FLOPs)

Term by term:

$N \times D$ — every one of the $D$ tokens passes through every one of the $N$ parameters, so the work scales as their product.
the $6$ — a bookkeeping constant: $2$ (each parameter does a multiply and an add) $\times 3$ (the backward pass that learns costs about twice the forward pass, so forward + backward $\approx 3 \times$ the multiply-add).

WHY it matters: this is our leash. $C$ is fixed the moment your boss hands you a budget. So $N$ and $D$ are not free — they are chained: raise one, the other must fall.

PICTURE:

Every point on the yellow curve $D = C / (6 N)$ is a legal choice: same compute, different split. Up-and-left = tiny model gorged on data. Down-and-right = huge model starved of data. Our job is to find the one point on this curve where $L$ is smallest.

Step 3 — WHY we can throw away the floor $E$

WHAT: We drop $E$ and only minimize the trade-off part:

f (N, D) = \frac{A}{N ^{α}} + \frac{B}{D ^{β}}

WHY: $E$ is a constant. Adding a constant lifts the whole landscape by the same height everywhere — it never moves the location of the lowest point. Like adding 10 cm to every mountain: the tallest peak is still the tallest peak, at the same coordinates.

PICTURE:

The two curves (with and without $E$ ) are identical in shape, just shifted up. The red dot — the minimum — sits at the same $N$ in both. That's the whole justification.

Step 4 — WHAT the constraint does: collapse two variables into one

WHAT: Use the leash $C = 6 N D$ to eliminate $D$ . Solve it for $D$ :

D = \frac{C}{6 N}

then substitute into $f$ :

f (N) = A N^{- α} + B (\frac{C}{6 N})^{- β} = shrinks as N ↑ A N^{- α} + grows as N ↑ B (6/ C)^{β} N^{β}

Term by term for the tricky second piece: $(\frac{C}{6 N})^{- β} = (\frac{6 N}{C})^{β} = (6/ C)^{β} N^{β}$ . The $(6/ C)^{β}$ is a fixed number (compute is fixed); the live part is $N^{β}$ , which grows with $N$ .

WHY: With the leash applied, $D$ is no longer independent — every $N$ forces a $D$ . So we have exactly one free knob, $N$ . One variable = plain single-variable calculus. This is the payoff of the constraint.

PICTURE:

Two opposing forces on one axis: the blue term falls (bigger model = less parameter-starvation), the yellow term rises (bigger model = less data left, more data-starvation). Their sum (green) is a valley. The bottom of that valley is the answer.

Step 5 — WHY a minimum exists, and WHERE (set the slope to zero)

WHAT: Find where the green valley bottoms out. The bottom of a valley is where the ground is momentarily flat — slope zero. The slope is the derivative $df / d N$ :

\frac{df}{d N} = pull down (negative) - α A N^{- α - 1} + push up (positive) β B (6/ C)^{β} N^{β - 1} = 0

WHY the derivative, and WHY this must be the minimum? The derivative is "slope of the sum curve". At the valley floor the two forces exactly balance — the downward pull equals the upward push, net slope $0$ . And we know it's a genuine minimum (not a fluke) because $f (N)$ is unimodal: the first term strictly falls and the second strictly rises on all $N > 0$ , so their sum falls, then rises, crossing flat exactly once. One crossing = one valley bottom. (See Overfitting vs underfitting — the two terms are literally underfitting-from-too-few-params vs the data-limited side.)

PICTURE:

The red tangent line is flat exactly at the valley floor. Left of it the slope is negative (still going down), right of it positive (climbing) — the sign flips once, confirming the single minimum.

Step 6 — WHAT the algebra gives: the scaling exponents

WHAT: Solve the balance equation. Move one term across:

α A N^{- α - 1} = β B (6/ C)^{β} N^{β - 1}

Collect all $N$ 's on one side by dividing both sides by $N^{β - 1}$ :

α A N^{- α - 1 - (β - 1)} = β B (6/ C)^{β} \Rightarrow N^{- α - β} \propto C^{- β}

(the exponent simplifies: $- α - 1 - β + 1 = - (α + β)$ ; and $(6/ C)^{β} \propto C^{- β}$ ). Flip both signs of the exponents:

N^{α + β} \propto C^{β} \Rightarrow N_{opt} \propto C^{β / (α + β)}

And since $D = C / (6 N)$ , whatever exponent $N$ takes, $D$ takes one minus it:

D_{opt} \propto C^{α / (α + β)}, \frac{β}{α + β} + \frac{α}{α + β} = 1

WHY the exponents sum to 1: because $C = 6 N D$ . If $N \propto C^{a}$ and $D \propto C^{b}$ , then their product $N D \propto C^{a + b}$ must scale like $C$ itself, forcing $a + b = 1$ . The whole result is just "split the exponent 1 between two players, in the ratio $β : α$ ."

Plugging Chinchilla's fit $α \approx 0.34, β \approx 0.28$ :

N_{opt} \propto C^{0.28/0.62} \approx C^{0.45}, D_{opt} \propto C^{0.34/0.62} \approx C^{0.55}

PICTURE:

On log-log axes both are straight lines whose slopes are the exponents — $0.45$ for $N$ , the slightly steeper $0.55$ for $D$ . Data climbs a touch faster: the model should stay a bit hungry.

Step 7 — Edge and degenerate cases (never skip these)

The formula hides some cliffs. Walk each one.

Case $α = β$ (equal exponents). Then $β / (α + β) = 0.5$ exactly, so $N \propto C^{0.5}$ and $D \propto C^{0.5}$ — the famous "square-root each" rule. This is the special case, not the general one. Chinchilla's $α \neq = β$ is why the real answer tilts to $0.45/0.55$ .

Case $N \to 0$ (no model). $A / N^{α} \to \infty$ : no parameters means infinite parameter-starvation. Off the chart, useless — the valley wall on the far left.

Case $D \to 0$ (no data). $B / D^{β} \to \infty$ : no experience means infinite data-starvation. The far-right valley wall. Both walls confirm the minimum lives strictly inside, never at an extreme.

Case $C \to \infty$ (infinite compute). Both terms $\to 0$ ; loss $\to E$ , the floor. You can approach perfection but never beat entropy. Scaling has a ceiling.

Case: all compute into $N$ (the pre-Chinchilla mistake). You slide to the bottom-right of Step 3's curve — huge $N$ , tiny $D$ . The $B / D^{β}$ term explodes even as $A / N^{α}$ shrinks. Net loss rises. This is exactly why the 280B Gopher lost to the 70B Chinchilla.

The figure marks all five cases on the loss-vs-split curve so you can see where each lands. (Note: Mixture-of-Experts changes the $C \approx 6 N D$ accounting because not all parameters fire per token — that's a different leash, a story for another page.)

The one-picture summary

One frame compresses the whole journey: the constraint curve $D = C / (6 N)$ (yellow), loss contours draped over it (blue), and the red dot where the contour just kisses the constraint — the compute-optimal point, sitting slightly toward the data side.

Recall Feynman retelling — the whole walkthrough in plain words

You have a fixed pile of arithmetic to spend ( $C$ ). You can spend it on a bigger brain (more knobs $N$ ) or on more reading (more tokens $D$ ) — but not both freely, because the pile is fixed and cost $\approx 6 N D$ . Being wrong ( $L$ ) comes from three things: a floor you can't fix ( $E$ ), too-small a brain ( $A / N^{α}$ ), and too-little reading ( $B / D^{β}$ ). Ignore the floor — a constant never moves the best spot. Use the fixed pile to turn "brain and reading" into a single dial: crank the brain up and reading auto-drops. One term falls, the other rises, so there's a valley — and the valley bottom is where their forces balance (slope zero). Solving that balance splits the growth between brain and reading in the ratio of their power exponents, and because cost is brain × reading, those two growth rates must add to one. The fit says brain grows like $C^{0.45}$ , reading like $C^{0.55}$ — nearly equal, reading a hair faster. Rule of thumb: about 20 tokens per parameter. Go all-in on brain and you starve it; that's exactly how a model four times smaller beat a giant.

Recall Quick self-check

Why do the $N$ and $D$ exponents against $C$ sum to 1? ::: Because $C = 6 N D$ , so $N \propto C^{a}$ and $D \propto C^{b}$ force $a + b = 1$ . Why can we drop $E$ before minimizing? ::: It's a constant; adding it shifts the whole loss up equally and never moves the minimum's location. What guarantees a single minimum? ::: One term strictly falls in $N$ , the other strictly rises, so their sum is unimodal — slope crosses zero exactly once.

Flashcards

Why does the substitution

D = C / (6 N)

help the derivation?

It uses the fixed budget to remove one variable, turning a two-variable minimization into single-variable calculus.

At the compute-optimal point, what balances?

The downward pull of the parameter term

- α A N^{- α - 1}

exactly cancels the upward push of the data term

+ β B (6/ C)^{β} N^{β - 1}

(slope zero).

What happens to loss as

C \to \infty

Both power-law terms vanish and

L \to E

, the irreducible floor — you approach but never beat data entropy.

When would

N

and

D

both scale as exactly

C^{0.5}

Only in the special case

α = β

; the fitted

α \neq = β

gives

0.45

and

0.55

instead.