Compute-data-parameter tradeoffs

WHY do we even care?

WHY $C \approx 6ND$ ? Each token requires roughly:

2 FLOPs per parameter for the forward pass (one multiply + one add per weight = 2 FLOPs).
~4 FLOPs per parameter for the backward pass (gradients w.r.t. weights and activations, ≈ twice the forward cost).

So total per token ≈ $6N$ FLOPs. Over $D$ tokens:

The optimization problem (Chinchilla)

The empirical scaling law (Hoffmann et al., "Chinchilla", 2022) fits:

Formula Loss as a function of size and data

$L(N, D) = \underbrace{E}_{\text{irreducible}} + \underbrace{\frac{A}{N^{\alpha}}}_{\text{too small}} + \underbrace{\frac{B}{D^{\beta}}}_{\text{too little data}}$

$E$ : entropy of language you can never beat (irreducible loss).
$A/N^\alpha$ : penalty for limited parameters — vanishes as $N\to\infty$ .
$B/D^\beta$ : penalty for limited data — vanishes as $D\to\infty$ .
Fitted (Hoffmann et al.): $\alpha \approx 0.34$ , $\beta \approx 0.28$ . Caution: these $\alpha,\beta$ are the exponents on $N$ and $D$ in the loss law above. The paper also reports the compute-optimal allocation exponents directly as $a = 0.46$ , $b = 0.54$ (so $N_{\text{opt}}\propto C^{0.46}$ , $D_{\text{opt}}\propto C^{0.54}$ ). Do not confuse these with the much smaller raw fit values $\alpha\approx0.076$ , $\beta\approx0.095$ that appear when the loss is written in bits with a differently-parameterized fit — those give the slope of loss vs. $\log N$ and are a distinct quantity. Below we use $\alpha\approx0.34,\beta\approx0.28$ so the derived $a,b$ match the paper's headline $\approx0.5$ each.

HOW to find the optimum — derive it

We minimize $L(N,D)$ subject to $6ND = C$ (fixed budget).

Step 1 — substitute the constraint. Why? To turn a 2-variable constrained problem into 1 variable. From $6ND=C$ : $D = \dfrac{C}{6N}$ .

$L(N) = E + \frac{A}{N^\alpha} + B\left(\frac{6N}{C}\right)^{\beta}$

Step 2 — differentiate and set to zero. Why? The minimum of a smooth curve has zero slope.

$\frac{dL}{dN} = -\alpha A N^{-\alpha-1} + \beta B\left(\frac{6}{C}\right)^\beta N^{\beta-1} = 0$

Step 3 — solve for $N$ . Why? We want how $N_{\text{opt}}$ scales with $C$ . Move terms:

$\alpha A N^{-\alpha-1} = \beta B\left(\tfrac{6}{C}\right)^\beta N^{\beta-1}$ $N^{-\alpha-1-\beta+1} = \frac{\beta B (6/C)^\beta}{\alpha A} \;\Rightarrow\; N^{-(\alpha+\beta)} \propto C^{-\beta}$ $\boxed{N_{\text{opt}} \propto C^{\frac{\beta}{\alpha+\beta}}}$

Step 4 — get $D$ . Since $D = C/(6N)$ , $D_{\text{opt}} \propto C / C^{\beta/(\alpha+\beta)} = C^{\alpha/(\alpha+\beta)}$ .

Figure — Compute-data-parameter tradeoffs

Worked examples

Common mistakes

Flashcards

What is the compute identity linking C, N, D?

C \approx 6ND

(2 FLOPs/param forward + 4 FLOPs/param backward, per token, over D tokens).

Why the factor 6 in

C=6ND

2 FLOPs/param forward + 4 FLOPs/param backward (grads w.r.t. weights and activations) = 6 FLOPs per parameter per token. (In FMAs it's 3, since 1 FMA = 2 FLOPs.)

Write the Chinchilla loss law.

L(N,D)=E+A/N^\alpha+B/D^\beta

: irreducible + parameter-limited + data-limited terms.

How does N_opt scale with compute C?

N_{opt}\propto C^{\beta/(\alpha+\beta)}\approx C^{0.46}

(Hoffmann et al.'s reported a≈0.46).

How does D_opt scale with compute C?

D_{opt}\propto C^{\alpha/(\alpha+\beta)}\approx C^{0.54}

(b≈0.54).

Why must exponents a and b sum to 1?

Because

C\propto ND

, so

N_{opt}D_{opt}\propto C^{a+b}=C^1

requires a+b=1.

Chinchilla rule of thumb for tokens per parameter?

~20 tokens per parameter.

Why was GPT-3 suboptimal?

Too large for its data: ~1.7 tokens/param vs optimal ~20; data-starved, over-parameterized.

When should you deviate from Chinchilla and over-train a small model?

When inference cost dominates — smaller model + more data is cheaper to serve.

Deriving N_opt: what constraint do you substitute?

D=C/(6N)

from the compute identity, then minimize L(N).

Recall Feynman: explain to a 12-year-old

Imagine you have a fixed amount of money to build and teach a robot. You can spend money on making the robot's brain bigger (more parameters) or on giving it more books to read (more data). If you build a giant brain but give it only one book, it's smart but knows nothing. If you give a tiny brain a million books, it can't remember them all. The trick is to balance: a bit bigger brain, a bit more books — about 20 pages for every bit of brain. That balance gives the smartest robot for your money.

Connections

Neural Scaling Laws — the power-law backbone this note fits.
Chinchilla vs GPT-3 — the empirical head-to-head.
FLOP accounting in Transformers — where "6" comes from precisely.
Inference cost vs training cost — why deployment changes the tradeoff.
Overfitting and capacity — the intuition behind data-starving.
Learning rate schedules — must match the token budget $D$ .

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Socho tumhare paas ek fixed compute budget hai — matlab kitne GPU-hours tum kharch kar sakte ho. Ye compute do cheezon mein bat-ta hai: model kitna bada hoga (parameters $N$ ) aur usko kitna data dikhaoge (tokens $D$ ). Inko jodne wala rule hai $C \approx 6ND$ . Yahaan "6" isliye aata hai kyunki har parameter par forward pass mein 2 FLOPs aur backward pass mein 4 FLOPs lagte hain, per token — total 6 FLOPs (dhyan rakho, ye FLOPs hain, FMA nahi; ek FMA = 2 FLOPs, toh FMA mein ye sirf 3 hote hain). Toh ye teeno — compute, data, parameters — ek triangle hai; do fix karo, teesra apne aap decide ho jaata hai.

Ab main sawal ye hai: fixed $C$ ke saath sabse kam loss kaise paayein? Loss ka formula hai $L = E + A/N^\alpha + B/D^\beta$ . Agar model bahut bada bana diya par data kam diya, to $B/D^\beta$ term badh jaayega — model data-starved ho jaata hai. Agar model chhota rakha par data zyada, to capacity saturate ho jaati hai. Isliye balance chahiye. Constraint $D=C/6N$ ko substitute karke, derivative zero karke solve karo, to nikalta hai $N_{opt}\propto C^{0.46}$ aur $D_{opt}\propto C^{0.54}$ — dono lagbhag barabar rate se badhte hain (ye Hoffmann et al. ke reported values hain).

Practical rule yaad rakho: ~20 tokens per parameter — ye Chinchilla-optimal hai. GPT-3 ne sirf ~1.7 tokens/param use kiye the, isliye wo over-parameterized tha; Chinchilla ne chhota (70B) model zyada data (1.4T tokens) par train karke usse beat kiya, wahi compute mein. Lekin ek twist: agar tum model ko billions users ko serve karoge, to inference cost dominate karega — tab jaan-boojh kar chhota model zyada data par "over-train" karna sasta padta hai (jaise LLaMA). Isliye rule tab badal jaata hai. Yaad rakho: dono ko saath badhao, sirf ek ko nahi.

WHY do we even care?

The optimization problem (Chinchilla)

HOW to find the optimum — derive it

Worked examples

Common mistakes

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Scaling & Efficient Architectures

Connections