Exercises — Neural scaling laws (Chinchilla, compute-optimal)

As $N\to \infty$, $A/N^{\alpha }\to 0$. As $D\to \infty$, $B/D^{\beta }\to 0$. Both reducible terms vanish, leaving $\boxed{E}$. It is the irreducible loss — the data's own entropy / Bayes error. No amount of brain or experience beats the inherent unpredictability of the text itself.

$C\approx 6ND$ FLOPs. The 6 $=2\times 3$:

• 2 = one multiply and one add per parameter per token (a multiply-add is two FLOPs).
• 3 = forward pass (1×) plus backward pass ($\approx 2\times$ forward), so $1+2=3$. Multiplying: $2\times 3=6$.

$D\approx 20N=20\times 7\text{B}=\boxed{140\text{B tokens}}$.

$C=6ND=6\times (70\times 10^9)\times (1.4\times 10^{12})$. $70\times 1.4=98$, so $C=6\times 98\times 10^{21}=588\times 10^{21}\approx \boxed{5.9\times 10^{23}\text{FLOPs}}$. (This is the Chinchilla-70B budget from the parent's worked example.)

$D/N=\frac{300\text{B}}{280\text{B}}\approx \boxed{1.07}$ tokens/param. Target is 20; Gopher sits at $\approx 1.1$, roughly 18× below the ideal. Yes — badly data-starved: far too much brain for far too little experience.

$N\to N\times 4^{0.45}$. Since $4^{0.45}=2^{0.9}\approx 1.87$, params grow $\approx \boxed{1.9\times }$. $D\to D\times 4^{0.55}$. Since $4^{0.55}=2^{1.1}\approx 2.14$, tokens grow $\approx \boxed{2.1\times }$. In words: roughly double both, with data getting a touch more of the growth. Check: $4^{0.45}\times 4^{0.55}=4^{1.0}=4$ ✓ — the two factors multiply back to the compute increase.

Substitute the two scalings into the compute law: $C\approx 6ND\propto C^a\cdot C^b=C^{a+b}.$ For $C\propto C^{a+b}$ to hold for all $C$, the exponents must match: $\boxed{a+b=1}$. The picture (figure s01, dashed line): the $\log N$ and $\log D$ growth-rate bars are two pieces of one whole. Chinchilla's $0.45+0.55=1.00$ ✓ — data simply takes the larger slice.

Differentiate (why: the minimum is where the slope crosses zero, and $f$ is unimodal — first term falls, second rises): $\frac{df}{dN}=-\alpha A{N}^{-\alpha -1}+\beta B(6/C{)}^{\beta }{N}^{\beta -1}=0.$ Move terms apart: $\alpha A{N}^{-\alpha -1}=\beta B(6/C{)}^{\beta }{N}^{\beta -1}.$ Divide both sides by $N^{\beta -1}$ (collecting powers: $N^{-\alpha -1-(\beta -1)}=N^{-\alpha -\beta }$): $\alpha A{N}^{-\alpha -\beta }=\beta B(6/C{)}^{\beta }.$ So $N^{-\alpha -\beta }=\frac{\beta B}{\alpha A}(6/C{)}^{\beta }$, i.e. $N^{\alpha +\beta }=\frac{\alpha A}{\beta B}(C/6{)}^{\beta }\propto C^{\beta }$. Take the $(\alpha +\beta )$-th root: $\boxed{N_{\text{opt}}\propto C^{\beta /(\alpha +\beta )}}$. With $\beta =0.28,\alpha +\beta =0.62$: exponent $=0.28/0.62\approx 0.45$ ✓.

The reducible loss is $0.30+0.05=0.35$. The param term ($0.30$) is 6× larger, so it is the dominant bottleneck. Spending on more params attacks the big term; spending on data only chips at the already-small $0.05$. Spend on $N$. In figure s02 the tall coral bar is the param-shortage term — you always push down the tallest bar first. This is exactly why balanced scaling wins: you keep both bars near equal so neither is a hidden bottleneck.

Doubling $N$: $A/(2N{)}^{\alpha }=0.10\times 2^{-0.34}=0.10\times 0.790=0.0790$. Doubling $D$: $B/(2D{)}^{\beta }=0.10\times 2^{-0.28}=0.10\times 0.824=0.0824$. New reducible loss $=0.0790+0.0824=\boxed{0.1614}$ vs old $0.20$. Drop $=\frac{0.20-0.1614}{0.20}=0.193\approx \boxed{19\%}$. Why balanced wins: both terms started equal and both shrank — no single bottleneck was left behind hogging the loss.

(a) Gopher: $C=6\times 280\text{B}\times 300\text{B}=6\times 8.4\times 10^{22}=5.04\times 10^{23}$ FLOPs. Chinchilla: $C=6\times 70\text{B}\times 1.4\text{T}=6\times 9.8\times 10^{22}=5.88\times 10^{23}$ FLOPs. Ratio $5.88/5.04\approx 1.17$ — Chinchilla used only ~17% more compute, essentially comparable. (b) Gopher's $D/N\approx 1.1$ makes its $B/D^{\beta }$ (data-shortage) term huge — it drowned in a lack of tokens. Chinchilla moved budget from params into tokens: the shed of $B/D^{\beta }$ far outweighed the small rise in $A/N^{\alpha }$ from the smaller model. Net: lower total loss. Smarter split, not just more FLOPs.

Use $D=20N$ in $C=6ND$: $C=6N(20N)=120N^2$, so $N=\sqrt{C/120}$. $N=\sqrt{5.9\times 10^{23}/120}=\sqrt{4.917\times 10^{21}}\approx 7.01\times 10^{10}=\boxed{70\text{B}}$. Then $D=20N=20\times 70\text{B}=\boxed{1.4\text{T}}$. This lands exactly on the real Chinchilla-70B design — the 20 tokens/param heuristic and the compute law together pin down $(N,D)$ from a single budget $C$.

$N_{\text{opt}}\propto C^{\beta /(\alpha +\beta )}=C^{0.5/1.0}=C^{0.5}$, and likewise $D_{\text{opt}}\propto C^{0.5}$. So $\alpha =\beta$ is exactly the condition that makes both exponents $0.5$. The famous "$\propto C^{0.5}$ each" is therefore the special case $\alpha =\beta$; Chinchilla's real fit ($\alpha =0.34\ne \beta =0.28$) breaks it, tilting to $C^{0.45}$ and $C^{0.55}$. The shortcut is a rounded approximation, not the truth.

At the optimum both terms fall at the same rate. $A/N^{\alpha }$ with $N\propto C^{\beta /(\alpha +\beta )}$ gives $N^{-\alpha }\propto C^{-\alpha \beta /(\alpha +\beta )}$. The same exponent appears for the data term. So $\boxed{\gamma =\frac{\alpha \beta }{\alpha +\beta }}$. Plug in: $\gamma =\frac{0.34\times 0.28}{0.62}=\frac{0.0952}{0.62}\approx 0.1535$. To halve loss: need $C^{-\gamma }$ to drop by 2, i.e. $C$-factor $=2^{1/\gamma }=2^{1/0.1535}=2^{6.51}\approx \boxed{91\times }$. Punchline: halving the reducible loss needs ~90× the compute — power-law scaling is brutally slow, which is why every fraction of the split matters.

Compute per token is driven by the parameters actually used on that token, so the FLOP law uses the active count: $C\approx 6N_{\text{act}}D=6\times 40\text{B}\times D$. This decouples capacity ($N=600$B, which helps the $A/N^{\alpha }$ representation term) from compute cost (set by $N_{\text{act}}=40$B). MoE therefore lets you buy a big-$N$ representation without paying full dense FLOPs — the dense Chinchilla $C\approx 6ND$ optimum is a lower bound these architectures try to beat by making $N>N_{\text{act}}$.

Compute-optimal scaling ::: split every compute increase so $N$ grows $\propto C^{0.45}$ and $D\propto C^{0.55}$ — roughly balanced, data slightly faster, exponents summing to 1.