Chain-of-thought prompting: Visual walkthrough

We assume you have never seen a probability formula. Every symbol below is earned before it is used.

Step 1 — What a language model actually does: one box, one guess

Picture a machine that is fed a strip of text tiles on the left and pushes out one new tile on the right.

Figure 1 (alt-text): four butter-yellow prompt tiles "23 / apples / use / 20" flow via a slate arrow into a lavender "MODEL (1 forward pass)" box, which emits a single mint tile "3" on the right. Caption: the model reads the whole prompt but writes just one new tile per forward pass.

The tiles going in are the prompt (your question).
The machine runs exactly one pass through itself — we will call that one pass of internal number-crunching a forward pass. Think of it as one "think".
Out comes one tile.

Step 2 — Why one tile is too little for a hard problem

Suppose the question is: "23 apples, use 20, buy 6 more — how many?" The true work is: subtract, then add. That's two operations. But if we force the model to answer in one tile — say the tile "9" — it must cram both operations into that one fixed forward pass.

Figure 2 (alt-text): a tall coral bar labelled "WORK REQUIRED (subtract, hold 3, then add)" stands beside a short mint bar labelled "BUDGET (1 forward pass, fixed height)"; a dashed line marks the budget ceiling and a coral note reads "OVERFLOW → guess (often 27)". Caption: when the work bar is taller than the fixed budget bar, the model overflows and guesses.

Left bar: the work required (two operations, some memory to hold "3" between them).
Right bar: the budget available in one forward pass (fixed height).
When the work bar is taller than the budget bar, the box overflows → it guesses. A common wrong guess here is "27" (it just adds the numbers it sees).

Step 3 — The trick: each new tile is a fresh forward pass, fed back in

Here is the move that changes everything. After the box writes a tile, that tile is appended to the input, and the box runs again — a brand-new forward pass — to produce the next tile.

Figure 3 (alt-text): four tiles "q → 3 → +6 → =9" laid left to right, joined by three coral arrows each labelled "pass 1", "pass 2", "pass 3". Caption: the tape grows one tile at a time; every arrow between tiles is a whole new forward pass.

Pass 1 reads the question, writes tile "3" (the result of $23 - 20$ ).
Pass 2 reads question + "3", writes the next reasoning tile.
Pass 3 reads question + "3" + ..., and so on until the stop token appears.

This is exactly the "scratchpad" idea from the parent note, but now you can see the mechanism: the tape grows tile by tile, each growth is one more forward pass, and it stops when the stop token is written.

Step 4 — Naming the pieces with symbols (earned at last)

Now that the picture is fixed, we attach names so we can write a formula.

Definition What

$p (something ∣ given)$ means — and are these real probabilities? $p (a ∣ q)$ reads: "the model's belief score that the answer is $a$ , given it saw the question $q$ ." The vertical bar $∣$ means "given / assuming". Higher = the model leans that way more. Picture: how strongly the box leans toward writing that tile. Are these genuine, calibrated probabilities? Yes in form, not always in trustworthiness. Internally the box produces raw scores called logits (one per candidate tile). These are turned into a proper probability distribution — every entry in $[0, 1]$ , summing to exactly $1$ over all candidate tiles — by the softmax step (Step 8). So $p (\cdot ∣ \cdot)$ is a normalised probability by construction. But "normalised" ≠ "well-calibrated": a model can assign $0.9$ to an answer that is right only $60%$ of the time. Treat $p (a ∣ q)$ as the model's stated confidence, which may be over- or under-confident on off-distribution problems — this mis-calibration is exactly why hard single-shot answers (Step 2) go wrong.

So:

$p (a ∣ q)$ — believe the answer straight from the question, no working shown. (Step 2's overflow situation.)
$p (r ∣ q)$ — believe a particular line of working, given the question.
$p (a ∣ r, q)$ — believe the answer given the working AND the question. (Step 3's read-the-scrapbook situation.)

Step 5 — The central equation, built term by term

We want to relate "answer straight away" to "answer after working." Two everyday facts about belief scores build the whole bridge.

Substitute Fact 1 into Fact 2 and you get the central result:

belief in answer with NO working shown p (a ∣ q) = r \sum answer given THIS working r p (a ∣ r, q) \cdot how likely working r is p (r ∣ q)

Reading it term by term:

$r \sum$ — the $\sum$ (capital Greek "S", for Sum) means "add over every complete reasoning chain $r$ ." The range: $r$ runs over all finite tapes $r_{1}, \dots, r_{k}$ for every length $k \in N_{0}$ (including the empty $k = 0$ chain) that the box could write before the stop token. This is a countably infinite set of sequences.
$p (a ∣ r, q)$ — for one fixed route $r$ , how sure is the model of the answer? Usually very sure, because it can just read the scrapbook.
$p (r ∣ q)$ — how likely was the model to take that route in the first place?
The dot $\cdot$ means multiply (Fact 1).

Formula Does summing over infinitely many chains even make sense?

Because $r$ ranges over infinitely many sequences, we should check the sum is well-defined. It is, for two reasons:

The route weights are a proper distribution. The model builds $r$ tile-by-tile, and at each position the next-tile softmax sums to $1$ (Step 4). Multiplying these per-tile probabilities down the chain, and summing over all chains that eventually hit the stop token, gives $\sum_{r} p (r ∣ q) = 1,$ provided the box halts with probability $1$ (every reasonable model has a non-zero chance of emitting the stop token at each step, so infinitely long chains have total probability $0$ ).
Bounded, non-negative terms. Each term $p (a ∣ r, q) p (r ∣ q)$ is $\geq 0$ and $\leq p (r ∣ q)$ . Since the $p (r ∣ q)$ already sum to $1$ , the whole series is dominated by a convergent series and therefore converges absolutely to a value in $[0, 1]$ . So $p (a ∣ q)$ is a genuine probability, and summing over $\infty$ -many working-out routes is completely legitimate.

Figure 4 (alt-text): a lavender dot "q" on the left; several coloured ribbons fan rightward, three thick mint ribbons converging on a mint node "a=9" and one thin coral ribbon reaching a coral node "a=27"; ribbon thickness represents p(r|q). Caption: the belief in an answer is the summed thickness of all ribbons that end on it — the marginalisation sum, drawn.

Step 6 — Following ONE path (greedy chain) all the way

Greedy decoding = keep only the thickest ribbon at each step, i.e. always write the single most-likely next tile. That traces one path from $q$ to $a$ , ending when the stop token is written.

Figure 5 (alt-text): four tiles rising like a staircase — butter "q", mint "r1: 23-20=3", mint "r2: 3+6=9", lavender "a: answer 9" — linked by coral arrows. Caption: greedy decoding traces one chain; each low step is an easy in-distribution hop, so nothing overflows.

$q$ : "23 apples, use 20, buy 6."
$r_{1}$ : " $23 - 20 = 3$ " — first hop. Each hop is an easy, in-distribution step.
$r_{2}$ : " $3 + 6 = 9$ " — reads the "3" it wrote, adds 6.
$a$ : "The answer is 9." (then the stop token — chain complete.)

Notice each arrow is a separate forward pass (Step 3) and each conditional $p (r_{2} ∣ r_{1}, q)$ is small and reliable. The staircase never overflows because no single step carries the whole problem.

Step 7 — Edge case: what if one path takes a wrong turn?

A single chain can slip. Suppose on one run the model writes $r_{1}$ : " $23 + 20 = 43$ " (wrong operation). Then every later hop is doomed — it computed correctly from a wrong start.

Figure 6 (alt-text): a lavender "q" dot with three mint ribbons converging on a mint node "9 (right)" and one lone coral ribbon reaching a smaller coral node "49 (wrong turn)". Caption: many valid routes pile onto the correct answer, while a wrong turn usually lands off on its own.

The green path is a correct route landing on 9.
The coral path took a wrong turn early and lands on a lone wrong answer (43+6 = 49).
Key observation: correct reasoning — however you phrase it — keeps converging on the same right answer, so many green ribbons pile up at 9; random slips tend to scatter to different wrong answers.

Common mistake Caveat — not every error is lonely: systematic errors can gang up

Why the naive claim feels right: "different mistakes → different wrong answers, so voting always saves us." The fix: this only holds for independent, random slips. If the model has a systematic bias — e.g. it consistently misreads "use" as "add", or a common misconception (like the "100 minutes" trap in the parent's Example 2) that many chains fall into the same way — then those wrong chains all converge on the same wrong answer. Now the wrong answer gets a fat pile of ribbons too, and majority voting can confidently return the wrong result. Self-consistency reduces variance (random noise); it cannot fix bias (a shared blind spot). So voting is a safeguard against scattered errors, not a guarantee of correctness.

Step 8 — Self-consistency: sample many paths, let them vote

To get different chains rather than the one greedy chain, we need randomness in which tile gets picked. That knob is temperature.

Definition Temperature (with the exact formula)

When the box is about to write a tile it holds a raw score, the logit $z_{i}$ , for each candidate tile $i$ . To turn logits into pick-probabilities we use the softmax with temperature $T$ : $p_{i} = \frac{e x p ( z _{i} / T )}{j \sum exp ( z _{j} / T )}$ Term by term:

$z_{i}$ — the logit (raw unnormalised score) of candidate tile $i$ .
$T$ — the temperature, a positive dial that divides every logit before exponentiating.
$exp (\cdot)$ — the exponential; it makes every value positive so the result is a valid probability, and it exaggerates gaps.
The denominator $\sum_{j} exp (z_{j} / T)$ — the normaliser: it forces $\sum_{i} p_{i} = 1$ (a proper distribution, exactly the calibration point from Step 4).

Reading the dial:

$T \to 0^{+}$ → dividing by a tiny number blows the top logit up to dominate → $p$ concentrates on the single best tile → greedy (the Step 6 staircase, same chain every run).
$T = 1$ → the model's native distribution.
$T > 1$ → gaps shrink, lower-ranked tiles get a real chance → diverse routes $r$ .

Picture: $T$ is a "surprise dial." At $0$ the box is a stubborn straight-liner; turn it up and side-paths open. (More in Temperature and Sampling.)

With $T > 0$ we ==sample $N$ diverse chains==. Each chain runs until its stop token (Step 4), giving its own answer $a_{i}$ . Then we count votes:

\overset{a}{^} = ar g a max i = 1 \sum N 1 [a_{i} = a]

Term by term:

$a_{i}$ — the answer from the $i$ -th sampled chain (there are $N$ of them).
$1 [a_{i} = a]$ — the indicator: it equals 1 if chain $i$ 's answer matches $a$ , and 0 otherwise. A vote-counter: "did this chain vote for $a$ ? yes=1, no=0."
$\sum_{i = 1}^{N} 1 [\dots]$ — add those up = how many chains voted for $a$ = the tally.
$ar g max_{a}$ — " $ar g max$ " means "the $a$ that makes the following biggest." So: pick the answer with the most votes. This $\overset{a}{^}$ (a-with-a-hat = our chosen estimate) is the winner.

Figure 7 (alt-text): three bars for answers "1/3", "1/2", "2/3" with heights 3, 1, 1; a lavender arrow points to the tallest, labelled "winner a-hat = 1/3". Caption: majority vote over five sampled chains selects the most-supported answer, 1/3.

Bars: answers 1/3, 1/2, 2/3 with vote counts 3, 1, 1 (the parent's Example 3).
The tallest bar is 1/3 → $\overset{a}{^} = 1/3$ .

Step 9 — Degenerate cases: when the staircase doesn't help

The one-picture summary

Figure 8 (alt-text): a left pair of mint budget bars (short then tall, arrow "grow tape = more passes") feeds a lavender "q" that fans mint ribbons up to a correct node and one coral ribbon down to a wrong node; on the right a mint vote bar towers over a coral one. Caption: the entire pipeline — short budget → grow the tape → many ribbons → biggest pile wins.

One image compressing the whole walkthrough: a fixed-height budget bar (one forward pass) is too short for a tall problem (Step 2); growing the tape adds passes and raises the budget (Step 3, capped by the context window); this traces ribbons through a fan of reasoning paths, summed by marginalisation (Step 5); many correct ribbons pile on the same answer while random errors scatter (Step 7); a majority vote reads off the peak (Step 8).

Recall Feynman retelling — the whole thing in plain words

An AI writes text one tile at a time, and for each tile it gets one burst of thinking — a fixed amount, no matter how hard the question — until it writes a special "I'm done" tile. For a two-step problem, one burst isn't enough, so it overflows and guesses (often wrong). The fix is sneaky: every tile it writes gets pasted back onto its input and earns it another burst. So if we let it write its working — "23 minus 20 is 3, plus 6 is 9" — each little tile buys another burst AND leaves a note it can read back later. One giant impossible leap becomes an easy staircase. (Careful: the scrapbook has a maximum size — the context window — so a chain that grows too long loses its start.) There are many correct ways to climb but each random mistake is its own weird detour; so if we turn up the "surprise dial" (temperature) and make it climb several times, then let the answers vote, the right answer usually wins the count — unless the model has a shared blind spot, in which case the wrong answer can win too. And voting isn't free: doing it $N$ times costs $N$ times the compute. Show your working, do it a few times, take the vote. That's chain-of-thought and self-consistency, drawn from the ground up.

Prompt Engineering · Few-shot In-context Learning · Self-Consistency Decoding · Tree of Thoughts · ReAct · Temperature and Sampling · Emergent Abilities of LLMs · Retrieval-Augmented Generation