Self-consistency and tree-of-thought: Question bank

Before we start, one plain-word reminder so no symbol surprises you:

A "chain" is one complete run of step-by-step reasoning ending in a final answer.
The "mode" is simply the answer that appears most often — the winner of a majority vote.
Temperature is the dial that controls how random the model's word choices are (higher = more varied); see Temperature and Sampling in LLMs.

Definition The symbols this page uses (read once, then nothing is a surprise)

$N$ = the number of reasoning chains we sample in self-consistency. Ask the model $N$ times, get $N$ answers, then vote.
$r$ = one reasoning chain (the hidden step-by-step working). We write $r^{(i)}$ for the $i$ -th sampled chain.
$a$ = the final answer a chain lands on (e.g. the number "24"). Different chains can give the same $a$ .
$p$ = the accuracy of a single chain: the probability that one chain, on its own, reaches the correct answer. So " $p = 0.6$ " means one chain is right $60%$ of the time.
$p (a, r ∣ prompt)$ = the joint probability that the model both takes reasoning $r$ AND lands on answer $a$ , given the prompt. Splitting it as $p (a ∣ r) p (r ∣ prompt)$ just says "first the model picks a reasoning $r$ , then that reasoning produces an answer $a$ ."
$p (a ∣ r)$ = the probability that reasoning $r$ yields answer $a$ . In the clean model where a chain's final answer is a deterministic read-off of its own working, this is == $1$ for the single answer that chain reaches and $0$ for every other answer== — so sampling a chain and reading its answer is a valid Monte-Carlo draw.
For Tree-of-Thought: $d$ = tree depth (how many reasoning steps deep we go), $k$ = branching factor (how many candidate next thoughts we generate per state), $b$ = beam/kept width (how many top-scoring states we keep at each depth after pruning).

True or false — justify

True or false: Self-consistency (SC) is a new kind of prompt you write.

False — SC changes only the decoding (sample many chains, then vote); the underlying CoT prompt is unchanged.

True or false: Running SC at temperature 0 still helps a little.

False — at temperature 0 the model is deterministic, so all

N

chains are identical and the vote has nothing diverse to combine; the benefit is exactly zero, not "a little."

True or false: If single-chain accuracy is below 50%, adding more chains and voting still raises accuracy toward 1.

False — the Condorcet result only pushes majority accuracy toward 1 when

p > \frac{1}{2}

; for

p < \frac{1}{2}

the majority actually drifts toward being wrong.

True or false: SC votes by averaging the models' token-level confidence numbers.

False — SC votes over discrete final answers. Two chains with totally different reasoning but the same answer each contribute one equal vote; confidence scores never enter.

True or false: Tree-of-Thought (ToT) is just SC with more samples.

False — SC samples complete independent chains and votes at the very end with no interaction; ToT builds partial thoughts, evaluates intermediate states, and can backtrack — it is a search algorithm, not a vote.

True or false: In SC the

N

chains talk to each other while generating.

False — the chains are drawn independently; independence is precisely what makes the majority-vote / Monte-Carlo argument valid.

True or false: ToT can solve problems that pure CoT cannot even in principle reach.

True in practice — dead-end tasks (puzzles, Game-of-24) need abandoning a committed step; linear CoT has no undo, so ToT's prune-and-backtrack unlocks paths a single chain gets stuck before.

True or false: SC is a Monte-Carlo estimate of the answer marginal

p (a ∣ prompt)

True — sampling

r^{(i)} \sim p (r ∣ prompt)

and counting matching answers estimates

p (a ∣ prompt) = \sum_{r} p (a ∣ r) p (r ∣ prompt)

; taking the mode is the

ar g max

of that estimate.

Spot the error

"To get diverse chains for SC I should raise temperature all the way to 2.0." — what's wrong?

Overshooting: too high a temperature makes the reasoning incoherent, dropping each chain's individual accuracy

p

possibly below

\frac{1}{2}

, which sabotages the vote. Moderate values (≈0.5–0.7) balance diversity against correctness.

"SC picks the chain with the highest probability." — what's wrong?

SC ignores per-chain probability entirely; it picks the answer that the most chains agree on. A rare-looking chain and a common one cast equal votes.

"ToT's evaluator must be a separate trained model." — what's wrong?

The state evaluator

V (s)

can be the same LLM prompted to judge ("sure / maybe / impossible") or even a cheap heuristic; no dedicated trained network is required.

"Because SC uses many chains, it must be more expensive than ToT." — what's wrong?

Reversed — SC costs about

N \times

a single call, while ToT costs roughly

d \cdot b \cdot k

generate plus evaluate calls (depth

\times

kept width

\times

branching), which is typically

≫ N

. ToT is the pricier one.

"In ToT, BFS and beam search keep every branch alive." — what's wrong?

They prune: beam/BFS keep only the top-

b

states per depth and drop the rest; keeping everything would explode the tree exponentially. See Beam Search and BFS/DFS.

"SC works because it reduces the variance of the LLM's token sampling." — what's wrong?

The real mechanism is that wrong paths scatter across many different wrong answers while correct paths converge on one, so the mode concentrates on the truth. It's about agreement of answers, not smoothing token noise.

"If two of five chains give answer A and the other three give three different answers, the vote is a tie." — what's wrong?

Not a tie — A has 2 votes and each other answer has 1, so A wins by plurality. Ties happen only when the top count is shared, e.g. 2 vs 2 vs 1.

Why questions

Why do we marginalize (sum) over the reasoning

r

instead of picking a best

r

We care only about the answer, so

r

is a nuisance variable; summing the joint

p (a, r)

over all

r

gives

p (a)

, and different reasonings that reach the same answer should reinforce that answer, not compete.

Why does the majority-correct probability climb toward 1 as

N

grows (when

p > \frac{1}{2}

Each chain is an independent noisy vote biased toward the truth; by the law-of-large-numbers/Condorcet logic the fraction correct concentrates above

\frac{1}{2}

, so the majority is almost surely correct as

N \to \infty

Why does ToT beat CoT on Game-of-24 style tasks?

The evaluate-and-prune step kills branches that can no longer reach the goal before wasting depth on them, so search recovers from bad first moves that a single greedy chain would be trapped in.

Why is SC called "orthogonal" to prompt engineering?

It touches only decoding, so you can improve the prompt AND add SC and stack both gains; neither replaces the other.

Why can't SC backtrack?

Its chains are generated whole and independently, then compared only at the end — there is no intermediate state to revisit, so nothing to back up from.

Why does raising diversity help SC but hurt if taken too far?

Some diversity is needed so chains explore different reasonings and the correct answer can win by agreement; too much diversity corrupts individual reasoning, dropping

p

so low that the vote loses its bias toward truth.

Edge cases

Edge case:

N = 1

chain with SC — what happens?

SC degenerates to ordinary CoT; with only one sample the "majority" is just that single answer, so there is no voting benefit.

Edge case: all

N

chains produce different answers (total scatter).

The mode has count 1 and SC essentially returns an arbitrary/tie-broken chain — a signal the problem is too hard or the sampling too noisy for voting to help.

Edge case: single-chain accuracy exactly

p = 0.5

The Condorcet lift vanishes — voting neither helps nor hurts in expectation; you're on the knife-edge where wrong answers can match right ones in frequency.

Edge case: ToT with branching

k = 1

and no pruning.

It collapses back into a single chain (a degenerate "tree" that is really a line) — no branching means no search advantage, just CoT.

Edge case: ToT's evaluator is perfectly accurate and prunes instantly.

Search becomes near-optimal — it follows only viable branches — but this is the ideal; real evaluators are noisy, which is why some branch width

b > 1

is kept as insurance.

Edge case: an even number of chains (say

N = 4

) and a 2–2 split.

A true tie occurs; a tie-break rule (random pick, or the lower-index answer) decides — one reason odd

N

is often preferred so majorities are cleaner.

Edge case: SC on a task with a continuous numeric answer where no two chains match exactly.

Exact-match voting fails because every answer is unique; you must bucket/round or cluster answers first (or use an evaluator) before a "mode" is meaningful.

Recall One-line self-test

Cover the answers and re-derive each "why." If you can state, for any trap, the mechanism (agreement of answers, independence, marginalization, prune-and-backtrack), you own the topic. See also Retrieval-Augmented Generation (RAG) for how these decoding ideas combine with retrieval.