Emergent abilities in large models: Worked examples

Before any symbol appears, here is the whole vocabulary, built from zero:

Definition The five symbols we will reuse (define once, use forever)

$p$ = the probability that one single token (one "word-piece") of the answer is correct. A number between $0$ and $1$ . Think: "how good is the model at getting one step right?"
$L$ = the number of tokens that must all be correct for the task to count as solved. A whole number ( $1, 2, 3, \dots$ ). Think: "how long is the chain?"
$N$ = the size of the model (parameters, as in Scaling laws (Kaplan, Chinchilla)). Bigger $N$ generally means bigger $p$ . We use $N$ whenever an example says "scale the model up" — that phrase means increase $N$ , which nudges $p$ upward.
$Acc$ = exact-match accuracy = fraction of times the entire answer is right = $p^{L}$ (all $L$ tokens correct, assuming each is independent). This is a harsh, all-or-nothing metric.
$TokenScore$ = a smooth, forgiving metric = the expected number of correct tokens = $L p$ . Unlike $Acc$ it gives partial credit: getting $9$ of $10$ tokens right scores $9$ , not $0$ . We meet it properly in Ex 7.

The symbol $p^{L}$ means " $p$ multiplied by itself $L$ times". Because $p < 1$ , multiplying it by itself shrinks it — and that shrinking is the whole story of emergence. By contrast $L p$ is just " $p$ scaled up by $L$ " — no shrinking, no drama.

The scenario matrix

Every worked example below fills one cell of this table. The columns are the case classes this topic can throw at you.

Cell	Case class	What is being stressed	Example
A	Low $p$ (small model)	$p^{L} \approx 0$ — "looks impossible"	Ex 1
B	High $p$ (large model)	$p^{L}$ jumps up — "suddenly solved"	Ex 2
C	Both endpoints together (the jump)	forecast the emergence gap	Ex 3 (figure)
D	Degenerate $L = 1$	single-token task never emerges	Ex 4
E	Limit $p \to 1$	ceiling behaviour, why it saturates	Ex 5
F	Limit $p \to 0$ / chance floor	why the curve sits flat at zero	Ex 6
G	Smooth-metric "mirage"	same model, no jump under a soft lens	Ex 7 (figure)
H	Chained sub-skills ( $p^{k}$ , CoT)	composition, why steps can hurt	Ex 8
I	Real-world word problem	3-digit arithmetic, pick $L$ yourself	Ex 9
J	Exam twist / inverse scaling	bigger model, worse score	Ex 10

We now walk them in order.

Ex 1 — Cell A: low $p$ , the "impossible" floor

Worked example Statement

A small model (small $N$ ) gets each token right with $p = 0.60$ on a task needing $L = 15$ tokens all correct. What exact-match accuracy do we expect?

Forecast (guess first): with $p = 0.6$ (which sounds "more than half right"), do you expect accuracy near $0.6$ ? Near $0.3$ ? Near $0$ ?

Steps

Write the metric. Why this step? Exact-match needs every token right, and the tokens are treated as independent, so the probability of all $L$ being right is the product $p \times p \times \dots = p^{L}$ . $Acc = p^{L} = 0.6 0^{15}$
Compute. Why this step? to feel how brutal the exponent is. $0.6 0^{15} \approx 0.00047$
Interpret. Why? $0.6$ sounds decent, but raised to the 15th power it is under $0.05%$ — indistinguishable from "can't do it at all". This is the flat floor of the emergence curve.

Verify: sanity check bounds. Since $0 < 0.6 < 1$ , we must have $0 < 0. 6^{15} < 0.6$ , and indeed $0.00047 < 0.6$ . ✓ The number is far below chance-visible levels.

Ex 2 — Cell B: high $p$ , the "suddenly solved" ceiling

Ex 3 — Cell C: the two endpoints together (forecast the gap)

Worked example Statement

Doubling parameters ( $N \to 2 N$ ) moves $p$ from $0.85 \to 0.97$ on a $L = 30$ task. Forecast the exact-match accuracy before and after, then measure the "emergence gap".

Forecast: guess both numbers and their ratio.

Figure below (what it shows): the horizontal axis is per-token correctness $p$ running $0$ to $1$ ; the vertical axis is exact-match accuracy $Acc = p^{L}$ , also $0$ to $1$ . Two curves are drawn: a coral curve for $L = 30$ (the long chain) and a mint curve for $L = 5$ (a short chain). Two lavender dots mark the before-scale point $(0.85, 0.008)$ and after-scale point $(0.97, 0.40)$ , and a butter-yellow band between them is the emergence gap.

Steps

Before-scale accuracy. Why? this is the "small model" point on the curve. $Acc_{before} = 0.8 5^{30} \approx 0.0076$
After-scale accuracy. Why? the "large model" point. $Acc_{after} = 0.9 7^{30} \approx 0.401$
Emergence gap. Why? the visible "jump" is the difference (or ratio). $0.401 - 0.0076 \approx 0.393, \frac{0.401}{0.0076} \approx 53 \times$
Read the figure. Why this step? On the axes just described (both $0$ to $1$ ), the coral curve ( $L = 30$ ) hugs the bottom axis for most of the $p$ -range then rockets up between $p = 0.85$ and $p = 0.97$ — the butter-yellow shaded gap between the two lavender dots is the emergence. The mint curve ( $L = 5$ ) climbs gently from the same axis: shorter chains emerge less sharply.

Verify: matches the parent note's own Forecast-then-Verify ( $0.8 5^{30} \approx 0.0076$ , $0.9 7^{30} \approx 0.401$ ). ✓

Ex 4 — Cell D: degenerate $L = 1$ , emergence disappears

Worked example Statement

A task needs only a single token ( $L = 1$ ) — e.g. answer "yes" or "no". Per-token skill rises $p = 0.60 \to 0.97$ (as $N$ grows). How sharp is the accuracy jump now?

Forecast: with the amplifier $L = 1$ , do we still get a dramatic jump?

Steps

Plug $L = 1$ . Why this step? $L$ is the amplifier; setting it to $1$ removes amplification. $Acc = p^{1} = p$
Evaluate endpoints. Why? to compare with the $L = 30$ case of Ex 3. $Acc_{before} = 0.60, Acc_{after} = 0.97$
Interpret. Why? accuracy just equals $p$ — it improves as smoothly as $p$ does. No emergence. This is the degenerate boundary of the whole phenomenon: single-token metrics never "look emergent" because there is nothing to raise to a big power.

Verify: $0.6 0^{1} = 0.60$ and $0.9 7^{1} = 0.97$ exactly; gap $= 0.37 =$ the raw per-token gain. ✓ No amplification, as predicted.

Ex 5 — Cell E: the limit $p \to 1$ (why it saturates)

Worked example Statement

A large model already sits at $p = 0.99$ on the $L = 20$ task. It is scaled again to $p = 0.999$ . How much accuracy is left to gain, and why does the curve flatten at the top?

Forecast: near-perfect per-token skill — is the remaining accuracy gain big or small?

Steps

Accuracy at $p = 0.99$ . Why? starting point near the ceiling. $0.9 9^{20} \approx 0.818$
Accuracy at $p = 0.999$ . Why? to see the diminishing return. $0.99 9^{20} \approx 0.980$
Limit reasoning. Why this step? As $p \to 1$ , $p^{L} \to 1^{L} = 1$ . The curve saturates: once you are past the knee, further per-token gains buy less and less accuracy because you are running out of room below $1$ . $lim_{p \to 1} p^{L} = 1$
Interpret. Why? emergence is sharp in the middle, flat at both ends — an S-shape. The top flat part is this $p \to 1$ saturation.

Verify: $0.9 9^{20} \approx 0.818$ (matches parent note's number) and $0.99 9^{20} \approx 0.980 < 1$ , gain $\approx 0.16$ — small, confirming saturation. ✓

Ex 6 — Cell F: the limit $p \to 0$ (the flat chance floor)

Worked example Statement

Explain, with numbers, why the accuracy curve sits dead flat near zero for a whole range of small-model sizes, on an $L = 25$ task, as $p$ crawls from $0.10$ to $0.30$ .

Forecast: $p$ tripled from $0.10$ to $0.30$ . Does accuracy triple?

Steps

Evaluate both. Why? to expose the floor. $0.1 0^{25} = 1 0^{- 25}, 0.3 0^{25} \approx 8.5 \times 1 0^{- 14}$
Compare to "visible on a plot". Why this step? Both are astronomically below any measurable accuracy (a benchmark has maybe thousands of examples, so anything under $\sim 1 0^{- 4}$ reads as exactly $0$ ).
Limit reasoning. Why? As $p \to 0$ , $p^{L} \to 0^{L} = 0$ . For small $p$ , $p^{L}$ is effectively zero for a long stretch of scale — this is the flat left arm of the S-curve. Progress is happening (from $1 0^{- 25}$ to $1 0^{- 14}$ , an eleven-order-of-magnitude gain!) but the harsh metric shows nothing. $lim_{p \to 0} p^{L} = 0$
Interpret. Why? this is exactly the "small models tell us nothing" trap — under exact-match they look identical, though they differ enormously. Use a smooth metric (Ex 7) to see the difference.

Verify: $0.3 0^{25} \approx 8.5 \times 1 0^{- 14}$ , and it is vastly larger than $1 0^{- 25}$ yet both round to $0$ on any real benchmark. ✓

Ex 7 — Cell G: the "mirage", same model under a smooth metric

Worked example Statement

Take the exact $p$ -values from Ex 3 ( $0.85 \to 0.97$ , $L = 30$ ). Instead of exact-match, score with the smooth metric $TokenScore = L p$ (expected number of correct tokens, defined in the vocabulary block). Does emergence survive?

Forecast: will the smooth metric also show a sudden jump?

Figure below (what it shows): the horizontal axis is again per-token correctness $p$ ( $0$ to $1$ ); the vertical axis is a normalised metric value on a shared $0$ -to- $1$ scale. The lavender curve is exact-match $p^{30}$ (which leaps near $p = 1$ ); the mint straight line is $TokenScore / L = p$ (a smooth diagonal). Dotted vertical guides sit at $p = 0.85$ and $p = 0.97$ , with coral dots on the lavender curve and butter-yellow dots on the mint line marking those two scales.

Steps

Compute the smooth metric before/after. Why this step? $TokenScore = L p$ is linear in $p$ — no exponent, no amplifier. $TokenScore_{before} = 30 \times 0.85 = 25.5$ $TokenScore_{after} = 30 \times 0.97 = 29.1$
Compute the fractional change. Why? to compare "jump size" against the $53 \times$ ratio of exact-match (Ex 3). $\frac{29.1}{25.5} \approx 1.14 (a 14% rise)$
Interpret. Why? Exact-match said $53 \times$ ; the smooth metric says $1.14 \times$ . Same model, same $p$ — the jump was manufactured by the metric. This is the Schaeffer "mirage" argument made concrete.
Read the figure. Why? on the axes just described, the lavender exact-match curve leaps upward near the right edge; the mint TokenScore line is a gentle straight diagonal — between the two dotted guides it barely rises.

Verify: $30 \times 0.85 = 25.5$ and $30 \times 0.97 = 29.1$ exactly; ratio $29.1/25.5 \approx 1.14$ . ✓

Ex 8 — Cell H: chained sub-skills $p^{k}$ (why CoT can hurt small models)

Worked example Statement

Chain-of-thought prompting adds $k = 8$ reasoning steps, each with per-step success $q$ . A small model has $q = 0.55$ ; a large model has $q = 0.95$ . Chain success is $q^{k}$ . Does adding the steps help or hurt each model?

Forecast: does decomposing the problem into 8 steps help the small model?

Steps

Small-model chain. Why this step? each extra step is another factor $< 1$ , multiplying the failure risk. $0.5 5^{8} \approx 0.0084$
Large-model chain. Why? same formula, higher $q$ . $0.9 5^{8} \approx 0.663$
Compute the ratio. Why this step? to quantify how many times better the large model's chain is — the raw ratio is the honest measure of the gap, just like the emergence ratio in Ex 3. $\frac{0.9 5 ^{8}}{0.5 5 ^{8}} = \frac{0.663}{0.0084} \approx 79 \times$
Compare & interpret. Why? the small model's 8-step chain collapses to under $1%$ — worse than it would do guessing a short answer — while the large model keeps $66%$ , a $\approx 79 \times$ advantage. This is why CoT helps big models and hurts small ones — the composition penalty $q^{k}$ dominates until $q$ is high.
Cross-link. Why? this is the same $p^{L}$ engine, now with $k$ reasoning steps instead of $L$ output tokens — see In-context learning where a latent task must be applied consistently.

Verify: $0.5 5^{8} \approx 0.0084$ , $0.9 5^{8} \approx 0.663$ ; the large chain is $\approx 79 \times$ the small chain. ✓

Ex 9 — Cell I: real-world word problem (pick $L$ yourself)

Worked example Statement

A model does 3-digit addition (e.g. $472 + 385 = 857$ ). The answer is 3 digit-tokens, and every digit including carries must be exactly right, so treat $L = 3$ . Per-digit correctness rises with scale ( $N$ ) from $p = 0.70$ to $p = 0.98$ . What accuracy is reported at each scale, and does this match the "0% then high" observation from the parent note?

Forecast: with only $L = 3$ , is the jump dramatic or mild?

Steps

Choose $L$ . Why this step? the word problem requires you to model the task; a 3-digit sum needs 3 output digits all correct. $L = 3$
Small-scale accuracy. Why? the "before" point. $0.7 0^{3} = 0.343$
Large-scale accuracy. Why? the "after" point. $0.9 8^{3} \approx 0.941$
Interpret & reconcile. Why? Here the jump is real but milder ( $0.34 \to 0.94$ ) because $L = 3$ is small. Reported " $\sim 0%$ then high" curves usually involve longer answers or more carries (effectively larger $L$ ), which is why they look flatter-then-sharper. Match the sharpness to the chain length.

Verify: $0.7 0^{3} = 0.343$ exactly, $0.9 8^{3} \approx 0.941$ ; and because $L = 3 < 15$ , the jump ( $\approx 2.7 \times$ ) is far gentler than Ex 3's $53 \times$ . ✓ Consistent with "longer chain ⇒ sharper emergence".

Ex 10 — Cell J: exam twist, inverse scaling

Worked example Statement (exam-style)

On some tasks the bigger model does worse — Inverse scaling. Suppose a bigger model (larger $N$ ) learns a tempting wrong pattern, so its per-token correctness actually drops from $p = 0.90$ (small) to $p = 0.75$ (large) on an $L = 10$ task. What happens to exact-match accuracy, and does $p^{L}$ still explain the shape?

Forecast: if $p$ drops from $0.90$ to $0.75$ , is the accuracy drop mild or amplified?

Steps

Small-model accuracy. Why this step? baseline before the harmful scaling. $0.9 0^{10} \approx 0.349$
Large-model accuracy. Why? after the harmful pattern is learned. $0.7 5^{10} \approx 0.056$
Compute the drop ratio. Why? to measure how much the amplifier magnifies the regression. $\frac{0.9 0 ^{10}}{0.7 5 ^{10}} = \frac{0.349}{0.056} \approx 6.2 \times$
Interpret. Why? The $p^{L}$ amplifier works in both directions: a $0.90 \to 0.75$ per-token drop becomes a $0.349 \to 0.056$ collapse (a $\approx 6.2 \times$ fall). So the same machinery that makes good improvements look emergent makes bad regressions look like sudden failures — this is why inverse scaling can also appear abrupt.
Reconcile with the parent note. Why? it warned "bigger is not always suddenly better"; this is the concrete counterexample.

Verify: $0.9 0^{10} \approx 0.349$ , $0.7 5^{10} \approx 0.056$ ; drop ratio $\approx 6.2 \times$ . Amplification is symmetric, as claimed. ✓

Recall

Recall Which matrix cell never shows emergence, and why?

Cell D ( $L = 1$ ) ::: With a single required token, accuracy $= p^{1} = p$ , so it tracks per-token skill smoothly — nothing is raised to a big power, so no amplification, no jump.

Recall Same

$p$ shift ( $0.85 \to 0.97$ ): why $53 \times$ under exact-match but only $1.14 \times$ under TokenScore? Exact-match is $p^{L}$ (non-linear amplifier) while TokenScore is $L p$ (linear). ::: The metric, not the model, creates the apparent jump — the "mirage".

Recall What does the

$p \to 0$ limit tell us about small models? $p^{L} \to 0$ , so a wide band of small models all read as $0%$ under exact-match even though their true $p$ differs by many orders of magnitude — use a smooth metric to see the real trend.