Visual walkthrough — Compute-data-parameter tradeoffs

We are hunting one thing: given a fixed pile of compute, how big should the model be and how much data should it eat? Everything below builds toward that.

---

Step 1 — Draw the three quantities as a single bar

WHAT: We picture compute as a bar of fixed length. Training work is spent per-parameter, per-token, so the bar's length is set by both $N$ and $D$ multiplied together.

WHY: Before optimizing anything we must see that $N$ and $D$ are not free — they are chained by the compute we can afford. A picture of one fixed-length bar makes that chain visible.

PICTURE: The red bar is $C$. Underneath, the same length is drawn as a grid of $N$ rows × $D$ columns × 6 — every little square is one unit of work.

---

Step 2 — The chain: raise one player, the other must fall

WHAT: We plot the curve $D=C/(6N)$ — every point on it spends exactly the full budget.

WHY: This single curve is the "menu" of legal choices. Any point below it wastes compute (you could afford more); any point above it is unaffordable. So the optimum must live on this curve — which is what lets us collapse two unknowns into one.

PICTURE: A hyperbola. As $N$ grows, $D$ slides down. The red dot is one legal choice; drag it and it must stay on the curve.

$\underset{\text{tokens}}{\underbrace{D}}=\frac{\stackrel{\text{fixed budget}}{\overbrace{C}}}{\underset{\text{const}}{\underbrace{6}}\cdot \underset{\text{params}}{\underbrace{N}}}$

---

Step 3 — What "good" means: draw the loss landscape

Term by term, right where each symbol lives:

• $E$ — the irreducible floor: language has inherent randomness (which word comes next is never certain). No amount of $N$ or $D$ removes it.
• $A/N^{\alpha }$ — shrinks as $N$ grows. $\alpha \approx 0.34$ controls how fast. A small model can't represent enough patterns → this term is large.
• $B/D^{\beta }$ — shrinks as $D$ grows. $\beta \approx 0.28$. Starve the model of data → this term is large (it hasn't seen enough; see Overfitting and capacity).

WHAT: We draw $L$ as a height over the $(N,D)$ plane — a bowl-shaped landscape sloping down toward "big model, lots of data" (bottom-right-far).

WHY: Optimization is geometry: we want the lowest point we can reach while standing on the budget curve. Seeing the surface makes the next step obvious.

PICTURE: Contour lines of $L$ (like a topographic map). Darker = lower loss. The red budget curve from Step 2 is drawn on top — we may only walk along it.

---

Step 4 — Walk the budget curve, watch loss dip and rise

WHAT: We collapse the 2-D landscape onto the 1-D budget curve. Substitute $D=C/(6N)$ into $L$, giving a function of $N$ alone: $L(N)=E+\frac{A}{N^{\alpha }}+B{\left(\frac{6N}{C}\right)}^{\beta }$

Notice what substitution did to the last term: $D$ is gone, replaced by $6N/C$. As $N$ climbs, $(6N/C{)}^{\beta }$ rises — that's the data-starvation penalty reappearing as a function of model size. That is the whole tension in one line.

WHY: A curve in one variable has an obvious lowest point. We turned a constrained 2-D problem into "find the bottom of a 1-D valley."

PICTURE: The red curve $L(N)$: falling on the left (parameter-limited), rising on the right (data-limited), with a clear minimum marked by the red dot.

---

Step 5 — Find the bottom: where the slope is zero

WHAT: Differentiate each term of $L(N)$ using the power rule ($\frac{d}{dN}{N}^p=p{N}^{p-1}$): $\frac{dL}{dN}=\underset{\begin{array}{c}\text{slope of the}\\ \text{"too small" term}\\ (\text{negative: it falls})\end{array}}{\underbrace{-\alpha A{N}^{-\alpha -1}}}+\underset{\begin{array}{c}\text{slope of the}\\ \text{"too little data" term}\\ (\text{positive: it rises})\end{array}}{\underbrace{\beta B{\left(\frac{6}{C}\right)}^{\beta }{N}^{\beta -1}}}=0$

The two slopes have opposite signs — one term pulls loss down, the other pushes it up. They cancel exactly at the bottom.

WHY: Setting this to $0$ is the mathematical statement of "the downward pull of adding parameters exactly balances the upward pull of losing data." That balance point is the optimum.

PICTURE: The same curve as Step 4 with its tangent line drawn at three spots: sloping down (left), sloping up (right), and perfectly horizontal (the red tangent) at the minimum.

---

Step 6 — Solve for how the optimum scales with compute

WHAT: Move the two balanced terms to opposite sides and collect powers of $N$: $\alpha A{N}^{-\alpha -1}=\beta B{\left(\frac{6}{C}\right)}^{\beta }{N}^{\beta -1}$

Divide both sides by $N^{\beta -1}$ (subtract exponents on the left): $N^{(-\alpha -1)-(\beta -1)}=N^{-(\alpha +\beta )}\propto C^{-\beta }$

The $C^{-\beta }$ appears because the right side carried a ${\left(6/C\right)}^{\beta }$. Now flip the exponent (raise both sides to the power $-\frac{1}{\alpha +\beta }$): $\boxed{N_{\text{opt}}\propto C^{\frac{\beta }{\alpha +\beta }}=C^a}$

And the data, straight from the budget curve $D=C/(6N)$: $D_{\text{opt}}\propto \frac{C}{C^a}=C^{1-a}=C^{\frac{\alpha }{\alpha +\beta }}=C^b$

Plugging $\alpha \approx 0.34,\beta \approx 0.28$: $a=\frac{0.28}{0.34+0.28}\approx 0.46,b=\frac{0.34}{0.34+0.28}\approx 0.54,a+b=1.$

WHY $a+b=1$: because $N_{\text{opt}}{D}_{\text{opt}}\propto C^{a+b}$ must equal $C^1$ (the budget only stretches linearly). It's a built-in sanity check — see Chinchilla vs GPT-3.

PICTURE: A log–log plot: $\log N_{\text{opt}}$ and $\log D_{\text{opt}}$ versus $\log C$, two straight lines with slopes $a\approx 0.46$ and $b\approx 0.54$ — nearly parallel, both near $\frac{1}{2}$. That near-parallelism is the headline: grow both roughly equally.

---

Step 7 — Edge cases: what the ends of the curve look like

Case A — $N\to 0$ (a vanishingly tiny model). The $A/N^{\alpha }$ term $\to \infty$: loss blows up. A model with almost no parameters cannot represent language no matter how much data you pour in. The curve shoots to the sky on the far left.

Case B — $D\to 0$ (almost no data), forced by $N\to \infty$ on a fixed budget. Then $B/D^{\beta }\to \infty$: loss blows up again. A giant model that reads nothing is useless. The curve shoots up on the far right. This is exactly the over-parameterized failure — GPT-3 lived out here at ~1.7 tokens/param.

Case C — limit $C\to \infty$. Both penalty terms $\to 0$, so $L\to E$: with infinite compute (split optimally) you approach, but never beat, the irreducible floor $E$.

PICTURE: The Step-4 valley redrawn with both walls labelled: left wall "model too small" ($N\to 0$), right wall "data-starved" ($D\to 0$), and a dashed floor at height $E$ that the whole curve sits above.

---

The one-picture summary

Everything on one canvas: the fixed-budget hyperbola $D=C/(6N)$ (black), the loss contours it cuts across (grey), and the single red point where the budget curve grazes its lowest contour — the Chinchilla optimum. The two red arrows show that when $C$ grows, that point slides diagonally up-and-right, so both $N$ and $D$ increase together (slopes $a\approx b\approx 0.5$).

Recall Feynman retelling — say it back in plain words

You've got a fixed amount of work you can do. You can spend it making the model bigger or letting it read more — but not both, because the total work is roughly $6\times \text{size}\times \text{reading}$. Make it too small and it's too dumb to model language; let it read too little and it never learns what it could have. So you slide along the "spend-it-all" line and watch the error dip: one direction fixes dumbness, the other fixes ignorance, and they cross at a lowest point. When you do the algebra (find where the error curve goes flat), the answer is beautifully symmetric — every time you quadruple your compute you should make the model about twice as big and show it about twice as much data. Roughly 20 words of text per knob in the model. That's the whole story, and it's why a smaller-but-well-fed Chinchilla beat the bigger-but-starved GPT-3 on the same budget.

---

Related: Compute-data-parameter tradeoffs · Neural Scaling Laws · Chinchilla vs GPT-3 · FLOP accounting in Transformers · Inference cost vs training cost · Overfitting and capacity · Learning rate schedules