Compute-data-parameter tradeoffs: Question bank

Before you start, recall the pieces in play (build them fresh so no symbol is unearned):

$N$ = number of parameters (how big the model is — count of tunable weights).
$D$ = number of training tokens (how much text it reads).
$C$ = total training compute in FLOPs (floating-point operations, the raw "work" done), tied together by $C \approx 6 N D$ .

Definition Notation used throughout this page

The loss law. The Chinchilla scaling law models test loss as $L (N, D) = E + \frac{A}{N ^{α}} + \frac{B}{D ^{β}}$

$E$ = irreducible loss — the entropy of language itself, the floor you can never beat.
$A / N^{α}$ = parameter-limited penalty: shrinks as the model grows. $A$ is just a fitted positive constant that sets its size; $α \approx 0.34$ is the fitted exponent controlling how fast it shrinks.
$B / D^{β}$ = data-limited penalty: shrinks as data grows. $B$ is a fitted positive constant; $β \approx 0.28$ is its fitted exponent.
$A, B, E, α, β$ are all empirically fitted numbers — measured by fitting real training runs, not derived from theory. See Neural Scaling Laws.

The subscript "opt". $N_{opt}$ and $D_{opt}$ mean the compute-optimal choices — the specific $N$ and $D$ that minimize $L$ while obeying the fixed budget $C = 6 N D$ . They are the "best split" of a given budget.

Big-number shorthand. $B = 1 0^{9}$ (billion), $T = 1 0^{12}$ (trillion). So $1.4 T = 1.4 \times 1 0^{12}$ tokens.

The figure below shows why balance matters: each penalty term is a curve, and the total loss has a single valley.

Figure — Compute-data-parameter tradeoffs

And here is the tradeoff itself — along the fixed-budget line $6 N D = C$ , loss dips to a minimum at one $(N, D)$ split:

True or false — justify

Every statement below is either true or false. The reveal gives the reason, which is the whole point.

TF1. "At a fixed compute budget, doubling the model size always lowers final loss."

False. Fixed

C = 6 N D

means doubling

N

halves

D

; past the optimum the data-starvation term

B / D^{β}

grows faster than

A / N^{α}

shrinks, so loss goes up.

TF2. "The irreducible loss

E

can be driven to zero with enough compute."

False.

E

is the entropy of the language itself — inherent unpredictability that no model or data amount can remove; only the

A / N^{α}

and

B / D^{β}

terms vanish as

N, D \to \infty

TF3. "Chinchilla and GPT-3 differ because Chinchilla used far more total compute."

False. Chinchilla used roughly the same compute as GPT-3 but re-split it (smaller

N = 70 B

, larger

D = 1.4 T = 1.4 \times 1 0^{12}

tokens) — a better allocation, not more budget. See Chinchilla vs GPT-3.

TF4. "The exponents

a

and

b

N_{opt} \propto C^{a}

D_{opt} \propto C^{b}

must sum to 1."

True. Because

C \propto N D

, we need

N_{opt} D_{opt} \propto C^{a + b} = C^{1}

, forcing

a + b = 1

; the derived

a = β / (α + β)

b = α / (α + β)

indeed sum to 1.

TF5. "Since

a \approx 0.46

and

b \approx 0.54

, data should grow slightly faster than model size as compute increases."

True.

b > a

(barely), so tokens outpace parameters marginally — but both grow near

C^{0.5}

, which is the real headline: scale them together.

TF6. "The factor 6 in

C = 6 N D

counts fused multiply-adds."

False. It counts FLOPs (2 forward + 4 backward per parameter per token). In fused multiply-adds it would be 3, since 1 FMA = 2 FLOPs. See FLOP accounting in Transformers.

TF7. "If Chinchilla-optimal is best, you should never train past 20 tokens/parameter."

False. That optimizes training loss per FLOP. If inference dominates lifetime cost, deliberately over-training a smaller model (more tokens/param) is better — see Inference cost vs training cost.

TF8. "A model with lower training loss will always have lower loss at deployment scale."

False (as a blanket claim). Training-optimal minimizes loss for the training budget; a model chosen for cheap inference trades a little training loss for far lower serving cost.

TF9. "The scaling law

L (N, D) = E + A / N^{α} + B / D^{β}

is a proven theorem."

False. The constants

A, B, E

and exponents

α, β

are an empirical fit to observed runs — remarkably robust, but not derived from first principles, and it can break outside the fitted range.

TF10. "Because

β < 1

, adding data gives diminishing returns."

True. Loss falls like

B / D^{β}

with

β \approx 0.28 < 1

, so each doubling of

D

helps less than the last — the curve flattens.

Spot the error

Each line contains a flawed argument. The reveal names the flaw.

SE1. "GPT-3 had 175B params and 300B tokens, so 300B > 175B means it had plenty of data."

The comparison is tokens-per-parameter, not raw totals:

300/175 \approx 1.7

, about 12× short of the ~20 target. Absolute sizes are meaningless without the ratio.

SE2. "

C = 6 N D

is exact, so I can predict wall-clock training time to the minute."

6 N D

is a leading-order estimate ignoring attention (scales with sequence length), embeddings, and hardware utilization. It's a planning tool, not a stopwatch. See FLOP accounting in Transformers.

SE3. "To find the optimal

N

, I maximize

L (N, D)

and set the derivative to zero."

We minimize loss, not maximize it — lower loss is better. The zero-derivative condition finds the bottom of the loss curve, not a peak.

SE4. "I substituted

D = C / (6 N)

then optimized over both

N

and

D

freely."

Once you substitute the constraint,

D

is no longer free — it's determined by

N

. You optimize over the single remaining variable

N

. Optimizing both again double-counts the budget.

SE5. "Raw scaling-law fits sometimes report

α \approx 0.076

, so

N_{opt} \propto C^{0.076}

Those tiny numbers are the slope of loss vs.

lo g N

in a differently-parameterized fit — a different quantity. The allocation exponent is

a = β / (α + β) \approx 0.46

, not

α

itself.

SE6. "A tiny model on infinite data reaches zero loss because

B / D^{β} \to 0

Only the data term vanishes; the parameter term

A / N^{α}

stays large for small

N

, plus irreducible

E

. Capacity, not data, becomes the bottleneck.

SE7. "Overfitting means the model memorized, so more parameters always overfit more."

At fixed compute, more parameters means less data per parameter — the failure mode is under-training / data-starvation, not classic memorization. Capacity and data must be balanced. See Overfitting and capacity.

SE8. "Since bigger models win benchmarks, scaling

N

is the priority; data is secondary."

Benchmark leaders often had huge budgets, conflating more compute with more parameters. At equal compute, balanced scaling beats a giant, data-starved model.

Why questions

WQ1. Why does substituting the constraint

6 N D = C

before differentiating make the problem easier?

It converts a two-variable constrained minimization into a one-variable unconstrained one, so a single derivative-equals-zero condition suffices.

WQ2. Why is the minimum of the loss curve found where its slope (derivative) is zero?

A smooth curve turns from falling to rising at its lowest point, and exactly there its instantaneous slope is zero — that's what "bottom of the valley" means.

WQ3. Why does the backward pass cost roughly twice the forward pass (

4 N

2 N

Backprop must compute gradients with respect to both the weights and the activations of each layer, roughly doubling the multiply-add work of the single forward pass.

WQ4. Why must model and data scale at nearly equal rates (

a \approx b \approx 0.5

The loss-law exponents

α \approx 0.34

and

β \approx 0.28

are close, so

a = β / (α + β)

and

b = α / (α + β)

are both near

0.5

; neither penalty term dominates, so

N

and

D

each grow near

C^{0.5}

. See Neural Scaling Laws.

WQ5. Why does the "20 tokens per parameter" rule flip when inference dominates?

20:1 minimizes training cost for a target loss. If the model serves billions of queries, a smaller model (cheaper per query) trained on more tokens amortizes better over its lifetime.

WQ6. Why can't you improve loss just by lowering the learning rate at fixed

N, D

The learning-rate schedule affects how well you reach the loss the data-and-size allow, not the floor

E + A / N^{α} + B / D^{β}

itself. It's a separate lever. See Learning rate schedules.

WQ7. Why does the constraint

a + b = 1

act as a sanity check on any fitted allocation?

Because compute is literally proportional to

N \times D

; any fit claiming

a + b \neq = 1

would violate the definition

C = 6 N D

and signals an error.

WQ8. Why is over-training a small model called "wasteful for training but smart for deployment"?

For a fixed training budget you'd reach lower loss by enlarging

N

; but deployment cost scales with

N

per query, so the small over-trained model wins once inference volume is large.

Edge cases

Boundary and degenerate scenarios the reader must not be surprised by.

EC1. What happens to

L (N, D)

N \to \infty

with

D

held finite?

The parameter term

A / N^{α} \to 0

, so loss floors at

E + B / D^{β}

— data-limited. Adding parameters beyond this buys nothing.

EC2. What happens as

D \to \infty

with

N

held finite?

The data term

B / D^{β} \to 0

, leaving

E + A / N^{α}

— capacity-limited. The model saturates; extra data is wasted.

EC3. If

N \to 0

(essentially no model), what does the law predict?

A / N^{α} \to \infty

, so loss blows up — a model with no capacity can't fit anything, consistent with intuition.

EC4. Both

N \to \infty

and

D \to \infty

together — what's the loss?

It approaches

E

, the irreducible entropy floor. That's the best any system can ever do; you can't beat the language's own randomness.

EC5. At zero compute (

C = 0

), what does

C = 6 N D

force?

At least one of

N

D

is zero — no trained model exists, so the law only describes runs with genuine positive budget.

EC6. If your data is finite (you run out of unique tokens), does more compute still help via bigger

N

Only up to a point — repeating data yields diminishing gains and eventual overfitting; the balanced-scaling recipe assumes fresh tokens are available. See Overfitting and capacity.

EC7. Two runs share the same tokens/param ratio but different absolute sizes — same loss?

No. The ratio fixes the split, but larger absolute

N

and

D

(more total compute) still push both penalty terms lower, giving lower loss.

EC8. What if attention FLOPs are large (very long sequences)? Does

C \approx 6 N D

still hold?

It underestimates, because attention cost grows with sequence length and isn't captured by the per-parameter count. Use

6 N D

as a floor and measure real FLOPs for long-context models.

Recall One-line survival summary

Fixed budget question ::: Split $C = 6 N D$ so tokens/param $\approx$ 20 (training-optimal); shift smaller-and-longer if inference dominates. The single most-missed trap ::: At fixed compute, bigger $N$ means smaller $D$ — "bigger is better" is false past the optimum.