Neural scaling laws (Chinchilla, compute-optimal)

WHY do scaling laws exist at all?

The empirical finding: loss follows a power law plus an irreducible floor.

L(N, D) \;=\; \underbrace{E}_{\text{irreducible}} \;+\; \frac{A}{N^{\alpha}} \;+\; \frac{B}{D^{\beta}}

$E$ = entropy of the data / Bayes error you can never beat.
$A/N^\alpha$ = error from too few parameters (model can't represent the function).
$B/D^\beta$ = error from too little data (model can't estimate what it could represent).
Chinchilla fit: $\alpha \approx 0.34$ , $\beta \approx 0.28$ (close, but not equal → data role slightly stronger at the optimum).

HOW to derive the compute-optimal split (from scratch)

Goal: minimize $L(N,D)$ subject to the budget $C = 6ND$ being fixed.

Step 1 — Drop the irreducible floor. $E$ is constant, so it doesn't affect where the minimum is. Minimize $f(N,D) = A N^{-\alpha} + B D^{-\beta}$ . Why this step? Constants don't move optima; we only care about the trade-off terms.

Step 2 — Substitute the constraint. From $C=6ND$ , write $D = \frac{C}{6N}$ :

f(N) = A N^{-\alpha} + B\left(\tfrac{C}{6N}\right)^{-\beta} = A N^{-\alpha} + B\,(6/C)^{\beta} N^{\beta}.

Why? One free variable now — pure single-variable calculus.

Step 3 — Differentiate, set to zero.

\frac{df}{dN} = -\alpha A N^{-\alpha-1} + \beta B (6/C)^{\beta} N^{\beta-1} = 0.

Why? $f(N)$ is unimodal on $N>0$ — the first term $A N^{-\alpha}$ strictly decreases and the second $\propto N^{\beta}$ strictly increases, so their sum has exactly one interior minimum where the slope crosses zero. (It need not be globally convex, but unimodality is enough to guarantee the stationary point is the minimum.)

Step 4 — Solve for $N$ vs $C$ . Rearranging (see VERIFY):

N^{\alpha+\beta} \;\propto\; C^{\beta} \quad\Rightarrow\quad \boxed{N_{\text{opt}} \propto C^{\,\beta/(\alpha+\beta)}}, \qquad \boxed{D_{\text{opt}} \propto C^{\,\alpha/(\alpha+\beta)}}.

Why? Because $D = C/(6N)$ , the exponents of $N$ and $D$ must sum to 1 (they multiply to give $C$ ).

Figure — Neural scaling laws (Chinchilla, compute-optimal)

WORKED EXAMPLES

Worked example 1. The Gopher / Chinchilla comparison

Gopher: $N = 280$ B params, $D = 300$ B tokens → ratio $D/N \approx 1.1$ tokens/param (data-starved!). Its compute is $C \approx 6\cdot 280\text{B}\cdot 300\text{B} \approx 5.0\times10^{23}$ FLOPs. Chinchilla chose $N = 70$ B, $D = 1.4$ T → ratio $= 20$ , with $C \approx 6\cdot 70\text{B}\cdot 1.4\text{T} \approx 5.9\times10^{23}$ FLOPs (about 17% more, not identical). Why this wins: the $B/D^\beta$ term was huge for Gopher. Moving the budget from params into tokens dropped that term far more than it raised $A/N^\alpha$ . Result: Chinchilla (4× smaller!) beat Gopher on almost every benchmark at comparable compute. Smarter split, not just more FLOPs.

COMMON MISTAKES (Steel-manned)

Flashcards

What is the Chinchilla loss functional form?

L(N,D)=E+A N^{-\alpha}+B D^{-\beta}

: irreducible floor plus two power-law terms for finite params and finite data.

Approx FLOPs to train a dense transformer?

C \approx 6ND

(2 for multiply-add × 3 for fwd+bwd).

Compute-optimal scaling of

N

with compute

C

N_{\text{opt}} \propto C^{\beta/(\alpha+\beta)} = C^{0.28/0.62}\approx C^{0.45}

Compute-optimal scaling of

D

with compute

C

D_{\text{opt}} \propto C^{\alpha/(\alpha+\beta)} = C^{0.34/0.62}\approx C^{0.55}

(data grows slightly faster).

Chinchilla rule of thumb for tokens per parameter?

About 20 tokens per parameter (

D\approx 20N

If compute increases 4×, how do you split it?

Roughly double

N

and double

D

(

4^{0.45}\approx1.9

4^{0.55}\approx2.1

Why was Gopher suboptimal vs Chinchilla?

Gopher was too large and data-starved (

D/N\approx1

); at comparable compute Chinchilla rebalanced to ~20 tokens/param with a 4× smaller model and won.

What does the term

E

represent?

Irreducible loss — data entropy / Bayes error you can never beat by scaling.

Why does loss follow a power law (not exponential)?

Diminishing returns: each doubling of

N

D

helps less; that constant multiplicative decay is exactly a power law.

Why do the

N

and

D

exponents against

C

sum to 1?

Because

C=6ND

;

N\propto C^a

and

D\propto C^b

force

a+b=1

Are the compute-optimal exponents exactly 0.5 each?

No — only if

\alpha=\beta

. The fitted values give ≈0.45 for

N

and ≈0.55 for

D

; 0.5 is a rounded shortcut.

Recall Feynman: explain to a 12-year-old

Imagine studying for a test. You can either grow a bigger brain (more parameters) or read more books (more data). You only have so many hours (compute). If you get a huge brain but read only one book, you're a genius with nothing to think about — wasteful. If you read a library but have a tiny brain, you forget most of it. Chinchilla found the sweet spot: grow your brain and your reading at almost the same speed (reading just a touch faster), about 20 pages of reading per bit of brain. Every time you get 4× more hours, make the brain about twice as big and read about twice as much.

Connections

Transformer architecture — the $6ND$ cost model assumes dense attention/MLP FLOPs.
Compute budgets & FLOPs — where $C\approx6ND$ comes from.
Kaplan scaling laws — earlier laws that over-weighted $N$ ; Chinchilla corrected the data term.
Overfitting vs underfitting — the $A/N^\alpha$ vs $B/D^\beta$ trade-off is capacity vs estimation error.
Mixture-of-Experts — breaks the simple $6ND$ assumption (sparse activation).
Learning rate schedules — must match the token budget $D$ for the law to hold.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, scaling laws ka core idea simple hai: aapke paas fixed compute hota hai (GPU × time), aur ye compute do cheezon mein bat-ta hai — model kitna bada hai (N parameters) aur kitna data padhaate ho (D tokens). Loss ka formula hai $L = E + A/N^\alpha + B/D^\beta$ : ek irreducible floor $E$ (jo kabhi nahi hataa sakte), plus do power-law terms. Agar N kam hai to model chhota dimaag, agar D kam hai to kam experience. Dono ko balance karna padta hai.

Chinchilla ne dikhaya ki purane bade models (jaise Gopher, 280B params) actually data-starved the — unhone params to badha diye par tokens kam rakhe, isliye $B/D^\beta$ term bada reh gaya. Comparable compute pe unhone chhota model (70B) liya par ~20 guna zyada data diya, aur wo jeet gaya. Note: Gopher aur Chinchilla ka compute bilkul same nahi tha (Chinchilla ~17% zyada), par point ye hai ki split smart tha, sirf zyada FLOPs nahi. Rule of thumb yaad rakho: ~20 tokens per parameter.

Derivation seedha hai: constraint $C = 6ND$ (ye 6 aata hai 2 multiply-add × 3 forward+backward se). Isko substitute karke minimize karo, to milta hai $N \propto C^{0.45}$ aur $D \propto C^{0.55}$ — dono ko saath mein grow karo, par data thoda tez ( $0.55 > 0.45$ ). Log aksar "dono $C^{0.5}$ " bolte hain — wo bas rounding hai, kyunki $\alpha\approx\beta$ hai par exactly equal nahi. Compute 4× ho gaya? To N aur D dono roughly double. Sirf model bada mat karo warna dobara data-starve ho jaoge.

Neural scaling laws (Chinchilla, compute-optimal)

WHY do scaling laws exist at all?

HOW to derive the compute-optimal split (from scratch)

WORKED EXAMPLES

COMMON MISTAKES (Steel-manned)

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Scaling & Efficient Architectures

Connections