Exercises — Compute-data-parameter tradeoffs

What they got right: the forward pass really is $\approx 2N$ FLOPs per token (one multiply + one add per weight = 2 FLOPs). What they missed: training also runs a backward pass to compute gradients, costing about $4N$ FLOPs per token (gradients w.r.t. weights and activations, roughly twice the forward cost). Correct constant: $2N+4N=6N$ FLOPs per parameter per token, so $C\approx 6ND$. Their $C=2ND$ counts inference-style forward-only compute, not training.

False. $E$ is the irreducible loss — the entropy of language itself. The two shrinking terms $A/N^{\alpha }$ and $B/D^{\beta }$ both vanish as $N\to \infty$ and $D\to \infty$, but $E$ has no $N$ or $D$ in it, so no amount of compute touches it. It is the floor every model asymptotes toward.

Why start from the ratio? It fixes the shape of the split so we have one unknown. Set $D=20N$. $C=6ND=6\cdot N\cdot 20N=120N^2.$ $N=\sqrt{\frac{C}{120}}=\sqrt{\frac{1.2\times 10^{21}}{120}}=\sqrt{10^{19}}\approx 3.16\times 10^9.$ So $N\approx 3.2\text{B}$ parameters, and $D=20N\approx 6.3\times 10^{10}=63\text{B}$ tokens. Check: $6\cdot 3.16\times 10^9\cdot 6.32\times 10^{10}\approx 1.2\times 10^{21}$ ✓.

Ratio: $D/N=200\text{B}/1\text{B}=200$ tokens/param. FLOPs: $C=6ND=6\cdot 10^9\cdot 2\times 10^{11}=1.2\times 10^{21}$ FLOPs. Verdict: Chinchilla-optimal is ~20 tokens/param; this run uses 200, i.e. 10× more data than "training-optimal." So the model is heavily over-trained (small model, lots of data). This is not "wrong" — it is exactly the deployment-optimal choice when you plan to serve the model a lot.

From $D=C/(6N)$:

• Run A: $D=\frac{6\times 10^{22}}{6\cdot 6\times 10^{10}}=\frac{6\times 10^{22}}{3.6\times 10^{11}}\approx 1.67\times 10^{11}=167\text{B}$. Ratio $=167/60\approx 2.8$.
• Run B: $D=\frac{6\times 10^{22}}{6\cdot 1.5\times 10^{10}}=\frac{6\times 10^{22}}{9\times 10^{10}}\approx 6.67\times 10^{11}=667\text{B}$. Ratio $=667/15\approx 44$. Optimal ratio is ~20. Run A (2.8) is too big/data-starved; Run B (44) is too small/over-fed. Both miss, but A is ~7× below optimum and B is ~2× above it — Run B is closer, so at equal compute Run B should reach the lower loss. (The true optimum is near $N\approx 22\text{B}$; see Example 2 in the parent.)

Why: the constraint $C=6ND$ says $N\cdot D\propto C$. If $N\propto C^a$ and $D\propto C^b$, then $N\cdot D\propto C^a\cdot C^b=C^{a+b}.$ For this to be proportional to $C^1$ (which the constraint demands), we need $a+b=1$. It is a conservation statement: the two growth rates must exactly split the compute exponent between them. Numeric: $0.46+0.54=1.00$ ✓. This is why "both grow at roughly equal rates $\approx 0.5$" — the budget forces them to share the exponent.

Step 1 — differentiate. What: take $dL/dN$. Why: the minimum of a smooth curve of loss vs. $N$ has zero slope. $\frac{dL}{dN}=-\alpha A{N}^{-\alpha -1}+\beta B{\left(\frac{6}{C}\right)}^{\beta }{N}^{\beta -1}.$ ($E$ is constant so its derivative is 0; the second term uses $N^{\beta }$ so its derivative brings down $\beta N^{\beta -1}$.) Step 2 — set to zero and separate. Why: find where slope $=0$. $\alpha A{N}^{-\alpha -1}=\beta B{\left(\frac{6}{C}\right)}^{\beta }{N}^{\beta -1}.$ Step 3 — collect powers of $N$. Divide both sides by $N^{\beta -1}$: $N^{-\alpha -1-(\beta -1)}=\frac{\beta B(6/C{)}^{\beta }}{\alpha A}\Rightarrow N^{-(\alpha +\beta )}=\frac{\beta B{6}^{\beta }}{\alpha A}{C}^{-\beta }.$ Step 4 — isolate $N$. Raise both sides to power $-1/(\alpha +\beta )$: $N={\left(\frac{\alpha A}{\beta B{6}^{\beta }}\right)}^{\frac{1}{\alpha +\beta }}{C}^{\frac{\beta }{\alpha +\beta }}\propto C^{\frac{\beta }{\alpha +\beta }}.■$ With $\alpha =0.34,\beta =0.28$: exponent $=0.28/0.62\approx 0.4516$, matching the reported $a\approx 0.46$.

$N_{\text{opt}}\propto C^{0.46}$, so $N$ grows by $8^{0.46}$; $D_{\text{opt}}\propto C^{0.54}$, so $D$ grows by $8^{0.54}$. $8^{0.46}=e^{0.46\ln 8}=e^{0.46\cdot 2.0794}=e^{0.9565}\approx 2.60,$ $8^{0.54}=e^{0.54\cdot 2.0794}=e^{1.1229}\approx \mathrm{3.07.}$ Model grows ~2.6×, data grows ~3.1×. Verify: $2.60\times 3.07\approx 8.0$ ✓ (since $8^{0.46}\cdot 8^{0.54}=8^1=8$). Neither dominates — both scale up, consistent with $a+b=1$.

(a) Training: $6ND=6\cdot 10^{10}\cdot 2\times 10^{11}=1.2\times 10^{22}$ FLOPs. Inference: $2N{D}_{\text{inf}}=2\cdot 10^{10}\cdot 5\times 10^{13}=1.0\times 10^{24}$ FLOPs. Ratio (inference / training): $\frac{1.0\times 10^{24}}{1.2\times 10^{22}}\approx 83$. Inference dominates by ~83×. (b) Why smaller $N$? Inference cost scales with $N$ linearly and forever ($2N{D}_{\text{inf}}$ every time you serve), while training pays $6ND$ only once. When inference dwarfs training, shrinking $N$ saves compute on every future forward pass. To keep loss low with a smaller model you compensate with more data (over-train), which is exactly the LLaMA-style choice. So the training-optimal 20-tokens/param is not the lifetime-optimal split — see Inference cost vs training cost.

Compute each term. With $N_P=2\times 10^{10}$: $N_P^{0.34}=e^{0.34\ln (2\times 10^{10})}$. $\ln (2\times 10^{10})=23.72$, so $N_P^{0.34}=e^{8.065}\approx 3186$. Thus $A/N_P^{\alpha }=400/3186\approx 0.1256$. $D_P=4\times 10^{11}$: $\ln =26.72$, $D_P^{0.28}=e^{7.482}\approx 1780$, so $B/D_P^{\beta }=300/1780\approx 0.1685$. $L_P=1.5+0.1256+0.1685\approx \mathrm{1.794.}$ Design Q: $N_Q=10^{11}$: $\ln =25.33$, $N_Q^{0.34}=e^{8.612}\approx 5490$, $A/N_Q^{\alpha }=400/5490\approx 0.0729$. $D_Q=8\times 10^{10}$: $\ln =25.11$, $D_Q^{0.28}=e^{7.031}\approx 1132$, $B/D_Q^{\beta }=300/1132\approx 0.2650$. $L_Q=1.5+0.0729+0.2650\approx \mathrm{1.838.}$ Design P wins ($1.794<1.838$): the balanced ratio-20 design beats the huge-but-starved design at (nearly) matched compute. This matches the intuition — Q spent too much on $N$ and starved $D$, inflating the $B/D^{\beta }$ term. See Overfitting and capacity.