Neural scaling laws (Chinchilla, compute-optimal): Question bank

Before we start, a one-line reminder of every symbol so nothing is used blind:

Three pictures to hold in your head

Before the traps, look at the shapes these questions keep referring to. The traps become obvious once you can see them.

Picture 1 — the falling-plus-rising curve (why there is a single best $N$ ). When you fix the compute budget $C$ , every choice of model size $N$ forces a data size $D = C / (6 N)$ . Small $N$ = tiny model, huge data → the model underfits (the $A / N^{α}$ term is big). Large $N$ = huge model, tiny data → the model is data-starved (the $B / D^{β}$ term is big). One term falls as $N$ grows, the other rises. Their sum is a valley with exactly one bottom.

Figure — Neural scaling laws (Chinchilla, compute-optimal)

Picture 2 — how loss falls with $N$ and with $D$ (why it's a power law). Each curve $A / N^{α}$ and $B / D^{β}$ is steep at first, then flattens toward the floor $E$ . Every doubling helps less than the last — that is the visual signature of a power law, not an exponential (which would keep falling at a constant fraction on a linear axis).

Picture 3 — the $6 N D$ FLOP accounting. Compute is not mysterious; it is bookkeeping. Follow the breakdown in the figure and the callout below.

A 4-line derivation of the exponents (so you never defer)

You do not need to leave this page to see where $C^{0.45}$ and $C^{0.55}$ come from.

True or false — justify

A bigger model always achieves lower loss than a smaller one.

False — only if data is also scaled. A giant model on tiny

D

leaves the

B / D^{β}

term huge and underlearns; a smaller well-fed model can beat it, which is exactly the Gopher-vs-Chinchilla story.

The irreducible loss

E

can be pushed to zero with enough compute.

False —

E

is the data entropy / Bayes error, the noise no model can predict away. Scaling

N

and

D

only shrinks the

A / N^{α}

and

B / D^{β}

terms, never

E

α = β

, then compute-optimal

N

and

D

would grow as

C^{0.5}

each.

True —

N

scales as

C^{β / (α + β)}

and

D

C^{α / (α + β)}

(note:

N

's exponent carries

β

on top,

D

's carries

α

). Setting

α = β

makes both equal

1/2

; it's precisely because

β < α

that

D

's exponent (0.55) exceeds

N

's (0.45) and data grows a touch faster.

The exponents of

N

and

D

against

C

must sum to exactly 1.

True — because

C = 6 N D

, if

N \propto C^{a}

and

D \propto C^{b}

then

C \propto C^{a + b}

, forcing

a + b = 1

. It's an accounting identity, not an empirical fit.

Since

α \approx 0.34

is "big", loss drops quickly as you add parameters.

False —

L \propto N^{- 0.34}

means halving the reducible loss needs

2^{1/0.34} \approx 7.6 \times

the params. A large exponent here still means slow, diminishing returns (see Picture 2's flattening curve).

Chinchilla beat Gopher mainly because it used far more compute.

False — compute was comparable (Chinchilla used ~17% more, not orders of magnitude). It won by rebalancing the same budget toward data (~20 tokens/param), not by brute force.

Using

C = N D

instead of

C = 6 N D

changes where the compute-optimal split lands.

False for the split, true for the number — the factor 6 is a constant multiplying

C

, so it scales

N_{o pt}

and

D_{o pt}

but the exponents

0.45/0.55

are unchanged; it only shifts the absolute FLOP count.

The loss curve

L (N)

at fixed

D

has an interior minimum you can differentiate to.

False — at fixed

D

L = E + A N^{- α} + const

is monotonically decreasing in

N

; more params always help. The interior minimum (Picture 1's valley) only appears once you couple

N

and

D

through the fixed budget

C

Spot the error

"To get 2× lower loss, just use 2× the parameters."

Error: loss falls as

N^{- α}

, not linearly. You need

2^{1/ α} \approx 7.6 \times

the params to halve the reducible part, and you never touch

E

at all.

"We fixed the budget, so we minimize

L (N, D)

freely over both

N

and

D

Error:

N

and

D

are not free — the constraint

C = 6 N D

ties them together. You substitute

D = C / (6 N)

first, turning it into one-variable calculus, then minimize.

E

is a constant, so we must keep it in the derivative when finding the optimum."

Error: a constant has zero derivative and cannot move the location of a minimum. You correctly drop $E$ and minimize only the two trade-off terms.

"When compute goes 4×, put it all into a 4× bigger model."

Error: that re-starves the model of data. Correct split:

N \times 4^{0.45} \approx 1.9

and

D \times 4^{0.55} \approx 2.1

— roughly double each.

"The heuristic

D \approx 20 N

is a law of physics for all models."

Error: it's a fitted rule of thumb for dense transformers at Chinchilla's regime, tied to specific

α, β

. Different architectures or objectives shift the optimal token/param ratio.

"Because both power-law terms shrink, doubling

N

and

D

roughly halves the loss."

Error: each term shrinks by only

2^{- α}

and

2^{- β}

(≈0.79, 0.82), and

E

doesn't move — the total loss drops modestly (~20% of the reducible part), not by half.

Why questions

Why does test loss follow a power law rather than an exponential decay?

Because each doubling of

N

D

gives a constant fractional improvement (diminishing returns). Constant multiplicative decay per doubling is the defining signature of a power law.

Why can we ignore the irreducible term

E

when locating the compute-optimal split?

Its derivative is zero, so it shifts the whole loss curve up-and-down without moving the horizontal position of the minimum — the location depends only on the trade-off terms.

Why is the optimum tilted slightly toward data (

C^{0.55}

) rather than exactly balanced?

Because the fitted

β < α

: data's exponent in the loss is smaller, so its term shrinks less per token, and the optimizer compensates by feeding it relatively more data.

Why is

C \approx 6 N D

and not just

N D

Per parameter per token you pay 2 FLOPs (one multiply + one add) on the forward pass, plus ~2× that on the backward pass —

2 + 4 = 6

. Bare

N D

forgets both the multiply-add factor and the backward pass.

Why did "bigger is always better" feel true in the GPT-3 era?

Those models grew

N

while holding

D

roughly fixed, so the

A / N^{α}

term was the visible bottleneck. Adding params genuinely helped then — but that hid the data-starvation waiting in

B / D^{β}

Why does the single-variable function

f (N)

have exactly one minimum?

After substitution it's a strictly decreasing term (

A N^{- α}

) plus a strictly increasing term (

\propto N^{β}

). A falling curve plus a rising curve is unimodal (Picture 1), so the slope crosses zero exactly once.

Edge cases

What happens to the compute-optimal recipe if your dataset is finite and you run out of unique tokens?

The

D = C / (6 N)

substitution assumes fresh tokens; once you must repeat data, extra epochs give diminishing (eventually harmful) returns and the clean power law breaks — you drift toward overfitting.

β \to 0

(data almost never helps), where does the compute-optimal split go?

N_{o pt} \propto C^{β / (α + β)} \to C^{0}

= constant, and

D_{o pt} \propto C^{α / (α + β)} \to C^{1}

— you'd freeze model size and pour everything into tokens. It's the mathematical limit of "data does all the work".

α \to 0

(params almost never help), what is the optimal strategy?

The exponents flip:

N_{o pt} \to C^{1}

D_{o pt} \to C^{0}

— grow the model with the whole budget and freeze data. This is the mirror image of the previous edge case.

Do these dense-transformer laws apply unchanged to a Mixture-of-Experts model?

No — MoE activates only a fraction of parameters per token, so the effective "active

N

" in

C \approx 6 N D

differs from total

N

. The

6 N D

accounting and the exponents must be re-derived for the sparse compute pattern.

How does the Chinchilla optimum differ from the earlier Kaplan scaling laws recommendation?

Kaplan's fit put more of extra compute into

N

(models grew faster than data); Chinchilla's more careful cosine-schedule fit found the balanced/data-tilted split — the disagreement traced to learning-rate schedule and fitting choices.

What does the loss

L (N, D)

approach as both

N \to \infty

and

D \to \infty

Both power-law terms vanish and

L \to E

, the irreducible floor. No finite or infinite scaling can cross below the data's intrinsic entropy.

At the compute-optimal point, are the two reducible terms

A / N^{α}

and

B / D^{β}

equal?

Not exactly equal. The optimality condition sets

α \cdot (A / N^{α}) = β \cdot (B / D^{β})

, so their ratio is fixed at

β / α \approx 0.82

— the data term sits at about 0.82× the param term at the optimum. They are comparable (neither is a lone bottleneck), which is why balanced growth wins, but not literally identical.

Recall Quick self-test

Cover the answers above; if you can justify each in one sentence with a mechanism (not "yes/no"), you own this topic.