Emergent abilities in large models: Visual walkthrough

Everything below assumes you know only what "multiply" and "a graph" mean. We earn the rest.

Step 1 — What is a "task", and what is the number we score it with?

WHAT. A language model reads a prompt and then emits tokens one at a time — think of a token as roughly one small chunk of text (a word or piece of a word). A task here means: "produce a specific answer that is a string of tokens."

WHY start here. Before any formula, we must agree on the thing being measured. The whole mystery of emergence lives in how we decide a task was solved. So we define the harshest common rule: exact match — the answer counts only if every single token is correct.

PICTURE. Below, an answer is a row of boxes. Each box is one token. A green box = that token came out right; a red box = wrong. Exact-match asks a yes/no question about the whole row: are they all green?

Figure — Emergent abilities in large models

Step 2 — The one honest number a model has: per-token correctness $p$

WHAT. Focus on a single box. Define $p$ as the probability that one token comes out correct. If $p = 0.9$ , then nine times out of ten that box lands green.

WHY this quantity. We want the smallest, most local measure of skill — one that can improve gently. Whole-answer accuracy is all-or-nothing and jumpy; a single token's chance of being right can slide smoothly. Choosing $p$ is choosing the lens that shows the truth underneath the illusion. (This is the same "smooth vs. sharp" tension the per-token loss captures.)

PICTURE. A single box with a dial from 0 to 1 underneath it. As the dial rises, green becomes more likely. The dial is $p$ .

Step 3 — Chaining boxes: where the multiplication comes from

WHAT. Exact-match needs box 1 right AND box 2 right AND ... AND box $L$ right. If we assume each box is an independent coin-flip with success chance $p$ , the probability that all land green is the product: $p \times p \times \dots \times p (L times) = p^{L} .$

WHY multiply, not add? "AND" for independent events means multiply their probabilities — that is the one rule of chained chances. Adding would answer a different question ("how many boxes on average," coming in Step 6). We multiply because exact-match is a chain that breaks at its weakest link.

Term by term: $whole-answer score Acc = one box’s chance p^{L how many boxes}$

PICTURE. A branching tree: at each box the path either stays "all-green so far" (probability $p$ ) or falls off into "already failed." Only the single all-green path survives to the end, and its width is $p^{L}$ .

Step 4 — What raising a number below 1 to a big power actually does

WHAT. Because $p$ is between 0 and 1, multiplying it by itself makes it smaller. The more boxes ( $L$ ), the more shrinking. So for a fixed imperfect $p$ , longer tasks are punished brutally.

WHY look at this. This is the engine of the "jump." We need to feel how violent $p^{L}$ is before we understand emergence. It is not the model being weird — it is the arithmetic of "a chain of maybes."

PICTURE. Curves of $Acc = p^{L}$ plotted against $p$ (horizontal), one curve per value of $L$ . For small $L$ the curve is gentle; for large $L$ it hugs the floor and then swings up almost vertically near $p = 1$ . The bigger $L$ , the sharper the elbow.

Step 5 — Why logarithms reveal the hidden smoothness

WHAT. Take the logarithm of both sides. A logarithm ( $lo g$ ) answers "what power do I raise the base to, to get this number?" Its magic here: it turns the multiplication-chain into a plain sum. $lo g Acc = lo g (p^{L}) = L \cdot lo g p .$

WHY use $lo g$ and not something else? We want to undo the exponent that is doing the deceiving. The log is the exact tool that converts " $p$ multiplied $L$ times" into " $L$ copies of $lo g p$ added" — a straight, honest line. It is the natural lens because it separates the task's contribution ( $L$ ) from the model's contribution ( $lo g p$ ).

Term by term: $log of whole-answer score lo g Acc = task length, fixed L \cdot model skill, slides smoothly lo g p$

PICTURE. Two panels stacked. Top: $Acc$ vs $p$ — the deceptive hockey-stick. Bottom: $lo g Acc$ vs $p$ — a smooth curve with no kink. Same data, two lenses; the jump lived only in the top lens.

Step 6 — The escape hatch: a smooth metric on the same model

WHAT. Change the question we score. Instead of "were all $L$ boxes green?" ask "how many boxes were green on average?" The expected count of correct tokens is $TokenScore = L \cdot p .$

WHY this matters. This metric is linear in $p$ — double $p$ and the score roughly doubles. No exponent, no amplification, no illusion. The identical model, viewed through this softer lens (like edit distance or log-likelihood), improves smoothly and predictably. The "emergence" was a property of the metric, not the mind.

Term by term: $avg green boxes TokenScore = how many boxes L \cdot one box’s chance, straight-line p$

PICTURE. Same box-row as Step 1, but now the score is a partial bar filling up proportionally to how many boxes are green — a ruler, not a light switch. Beside it, its graph vs $p$ is a straight line.

Step 7 — Plugging in the scaling law: where $p$ comes from

WHAT. Where does $p$ actually get its smoothness? From the Chinchilla loss law. Loss $L (N, D)$ (careful: this $L (\cdot)$ is loss, a different beast from token-count $L$ — we write it with arguments) falls smoothly as the model grows: $L (N, D) = E + \frac{A}{N ^{α}} + \frac{B}{D ^{β}} .$

Link loss to per-token correctness with $p (N) = e^{- (L (N, D) - E)}$ . Substitute into $Acc = p^{L}$ : $Acc (N) = exp (- L (L (N, D) - E)) .$

WHY. This shows the jump is a composition of smooth functions — mathematically continuous, yet the multiplier $L$ (task length) blows up the tiny residual loss into a visible cliff.

Term by term (the loss law):

$E$ = irreducible entropy — the noise floor no model can beat.
$A / N^{α}$ = the part you buy down with more parameters $N$ .
$B / D^{β}$ = the part you buy down with more tokens $D$ .

PICTURE. Left axis: loss gliding down a smooth power-law slope as $N$ grows. Right axis: the same $N$ mapped through $Acc (N)$ — flat, then a cliff. One smooth input, one dramatic output.

The one-picture summary

This final figure stacks the whole story: a smooth loss (Step 7) → smooth per-token $p$ (Step 2) → the amplifier $p^{L}$ (Steps 3–4) → the deceptive accuracy cliff (top), with the smooth $L \cdot p$ ruler (Step 6) drawn faintly behind it to remind you the model never jumped.

Recall Feynman retelling of the whole walkthrough

Picture a task as a row of 20 light-bulbs; you only "win" if every bulb lights. Each bulb lights with some chance $p$ . When you're weak, each bulb is on 80% of the time — but all twenty at once almost never happens, so your win-rate is basically zero. Now you get a little better, each bulb on 99% of the time. Suddenly all twenty light most tries, and your win-rate leaps to 82%. You didn't gain a magic power; every bulb just improved a hair, and because the game demands all of them, that tiny per-bulb gain looks like a switch flipping. If instead you scored "how many bulbs are lit on average," you'd see the truth: a smooth, steady climb. Big models are the bulbs; the smooth scaling law is your steady practice; and the harsh all-or-nothing scoreboard is what makes smooth learning look like sudden magic.

Recall

Why does exact-match accuracy use $p^{L}$ and not $L \cdot p$ ? ::: Exact-match needs every token right (an AND-chain of independent events), so probabilities multiply → $p^{L}$ . $L \cdot p$ is the expected count of correct tokens, a different (smooth) question. What does taking $lo g$ of $Acc = p^{L}$ reveal? ::: $lo g Acc = L lo g p$ , a straight-line, kink-free function of the model's skill — the jump was only in the un-logged lens. In one phrase, what is emergence mathematically? ::: Amplification of a small, smooth per-token gain by a large exponent $L$ — not a true discontinuity. Which single number in $Acc = p^{L}$ makes the transition sharper as it grows? ::: The task length $L$ .