Speculative decoding: Question bank — AI-ML deep dive

This page assumes the parent note vocabulary: the draft model $M_{q}$ (small, fast, proposes tokens with probability $q$ ), the target model $M_{p}$ (large, slow, the one whose output we actually want, probability $p$ ), the lookahead $K$ (how many tokens we guess ahead), and $α$ (the acceptance rate — the chance a single drafted token survives verification).

True or false — justify

Speculative decoding changes the text the target model would have produced.

False. The acceptance–resampling rule is designed so the final output distribution equals

p

exactly — it is lossless. It changes speed, not what is sampled.

A better (more accurate) draft model always gives a bigger speedup.

Mostly true but with a catch: what matters is agreement

α

with the target, not the draft's standalone task accuracy. A draft that is very accurate but slow can erase the gain, since draft time

K c_{q}

eats into every cycle.

If the draft model is identical to the target (

q = p

), every token is accepted.

True. The acceptance probability

min (1, p / q) = min (1, 1) = 1

for every candidate, so nothing is ever rejected — but you also gain nothing, because the "draft" is now as slow as the target.

The maximum possible speedup is exactly

K + 1

False.

K + 1

is only the idealized ceiling reached when the draft is free (

c_{q} \to 0

) and

α \to 1

. With real draft cost the true ceiling is

\frac{c _{p} ( K + 1 )}{K c _{q} + c _{p}} < K + 1

Increasing

K

always increases the speedup.

False. Larger

K

helps only while acceptances are likely. Once you pass the point where later drafted tokens are usually rejected, the extra draft passes cost time for tokens that get thrown away, and speedup falls.

Speculative decoding needs the draft and target to share the same tokenizer/vocabulary.

True. Acceptance compares

p (\tilde{x})

and

q (\tilde{x})

for the same token symbol

\tilde{x}

. If vocabularies differ, "the same token" isn't well-defined and the ratio

p / q

is meaningless.

If a candidate is rejected at position

t + i

, the already-accepted tokens before it are also thrown away.

False. Everything before the first rejection is kept. Only the rejected token and all candidates after it are discarded, because those later candidates were conditioned on a token we no longer trust.

The target model runs once per accepted token.

False. The target runs once per cycle — a single parallel forward pass verifies all

K

candidates at once. That parallel pass is the whole source of the savings.

Spot the error

"Accept the draft token with probability

min (1, q / p)

Ratio is upside down. The correct rule is

min (1, \frac{p ( x ~ )}{q ( x ~ )})

— you accept freely when the target likes the token at least as much as the draft did.

"On rejection, resample the replacement token straight from

q

Wrong — resampling from

q

would just re-inject the draft's bias. You resample from the adjusted distribution

\frac{max ( 0 , p - q )}{Z}

, which subtracts out what the draft over-claimed.

"Expected accepted tokens

= \sum_{i} i α^{i - 1} (1 - α)

Two errors: the exponent should be

α^{i}

(accepting

i

tokens costs

α^{i}

, not

α^{i - 1}

), and the token count is

i + 1

not

i

(the resampled token counts too). Correct closed form:

\frac{1 - α ^{K + 1}}{1 - α}

"Autoregressive decoding is slow because it is compute-bound."

Backwards. Single-token generation is memory-bound — the GPU spends its time streaming billions of weights in, not doing FLOPs. That idle compute is precisely what speculative decoding fills.

"If all

K

drafts are accepted, we've produced

K

tokens this cycle."

You get

K + 1

: after accepting all

K

, the target's own final pass already gives you a free bonus token sampled from

p (\cdot ∣ x_{< t + K})

"Because it's lossless, speculative decoding can never be slower than plain decoding."

Almost — but the draft passes have a small fixed cost. If

α

is tiny, you pay

K c_{q}

per cycle for tokens you mostly reject, so throughput can dip slightly below the baseline. It's lossless in output, not guaranteed faster.

"Distillation makes a good draft by maximizing the draft's task accuracy."

The distillation loss minimizes

KL (p ∥ q)

— it matches distributions, which is what raises

α

. Raw task accuracy and token-level agreement are different things; a 95%-accurate model can still agree with

p

only 60% of the time.

Why questions

Why does verifying

K

tokens cost the target roughly the same time as generating one?

Because verification is a single forward pass over

K

tokens, and that pass is compute-bound — the expensive part (loading weights) happens once, and adding a few more positions barely moves the memory-bound wall clock.

Why break (stop) at the first rejection instead of independently checking each candidate?

Each candidate

\tilde{x}_{t + i}

was drafted conditioned on the previous drafts being right. Once one is rejected, the context those later drafts assumed no longer holds, so their probabilities are invalid — you must re-draft from the corrected prefix.

Why is the acceptance rule

min (1, p / q)

and not just "accept if

p > q

A hard threshold would distort the distribution. The probabilistic rule, combined with the resample-on-reject step, is exactly what makes the combined output distribution collapse to

p

— a deterministic cutoff would not be lossless.

Why does resampling use

max (0, p - q)

rather than

p - q

For tokens where

p < q

, the difference is negative and can't be a probability. Those tokens were already handled correctly by the acceptance step; the resample distribution only needs the residual mass where the target wanted more than the draft supplied — hence the clip at zero.

Why do people use knowledge distillation (5.3.2-Knowledge-distillation) to build the draft rather than just picking any small model?

Speedup depends on

α

, and

α

is high only when

q

mimics

p

's shape. Distillation directly trains

q

to imitate

p

's output distribution, which is the exact quantity the acceptance step measures.

Why can't beam search (3.2.4-Beam-search) replace the draft-and-verify idea for the same speedup?

Beam search still runs the full target model at every step to expand candidates — it explores more paths but does not skip target forward passes. Speculative decoding's win comes from a cheaper model doing the guessing, which beam search never introduces.

Edge cases

K = 1

(guess only one token ahead).

Each cycle drafts one token and verifies it. Expected tokens

= \frac{1 - α ^{2}}{1 - α} = 1 + α

, so up to 2 tokens per target pass — a valid but modest special case, useful when memory or the draft cost is tight.

α \to 1

(draft is essentially perfect).

Nearly every candidate plus the bonus token is accepted, so output rate

\to K + 1

tokens per cycle and speedup approaches its ceiling

\frac{c _{p} ( K + 1 )}{K c _{q} + c _{p}}

α \to 0

(draft is almost always wrong).

You reject at the first token almost every cycle, producing ~1 token (the resampled one) while still paying the

K c_{q}

draft cost. Speedup

\to \frac{c _{p}}{K c _{q} + c _{p}} \leq 1

— no gain, mild overhead. Still lossless.

The draft samples a token that the target assigns probability

p = 0

Acceptance probability

min (1, 0/ q) = 0

, so it is always rejected. The target simply never wanted that token, and the resample step draws a legitimate replacement — correctness is preserved.

The very first drafted token is rejected.

You still make progress: the resample from

max (0, p - q) / Z

yields one valid target-distributed token. That cycle behaves like ordinary autoregressive decoding — the guaranteed floor of the method.

Draft and target both greedily agree, but you're doing greedy (temperature 0) decoding.

Then acceptance is deterministic: accept iff the draft's argmax equals the target's argmax at that position. The

p / q

machinery degenerates into a plain equality check, and the method is still lossless with respect to greedy output.

Recall One-line self-test

State the acceptance probability and what happens on the first rejection. ::: Accept $\tilde{x}_{t + i}$ with prob $min (1, p (\tilde{x}_{t + i}) / q (\tilde{x}_{t + i}))$ ; on the first rejection, resample one token from $max (0, p - q) / Z$ and discard all later candidates.