Classifier-free guidance: Visual walkthrough — AI-ML deep dive

Step 1 — What is the model even predicting? An arrow.

WHAT. A diffusion model, at some noisy step, is handed a fuzzy image $x_{t}$ and must answer one question: "which way is the noise?" Its answer is a vector — a list of numbers, one per pixel-channel — that we can picture as a single arrow living in the space of images.

WHY an arrow. Because the only thing we ever do with the model's output is subtract it to get a cleaner image. Subtracting a vector = stepping in a direction. So the output is a direction, and directions are arrows. Nothing more mysterious than that.

PICTURE. Below, one dot is our noisy image $x_{t}$ . The arrow $ϵ_{c}$ (magenta) is what the model predicts when it can read your prompt $c$ . Read " $ϵ$ " as "epsilon", the Greek letter we use for noise.

Definition The two arrows (and the timestep

$t$ )

$t$ — the diffusion timestep: which rung of the noising ladder we are on. Large $t$ = very noisy (almost pure static), small $t$ = almost clean. The model is called once per timestep as we walk from noisy down to clean; see DDPM sampling. On these figures $t$ is fixed at one such rung, so we just carry it along.
$ϵ_{θ} (x_{t}, t, c)$ — noise the model predicts knowing your prompt $c$ (at image $x_{t}$ , timestep $t$ ). Shortened to $ϵ_{c}$ .
$ϵ_{θ} (x_{t}, t, \emptyset)$ — noise it predicts ignoring the prompt (the symbol $\emptyset$ , "empty set", means no conditioning). Shortened to $ϵ_{\emptyset}$ .
The subscript $θ$ (theta) just says "these numbers come from the trained network's weights $θ$ ." One network produces both arrows — see Conditional generation.

The shorthands $ϵ_{c}$ and $ϵ_{\emptyset}$ always stand for the full $ϵ_{θ} (x_{t}, t, \cdot)$ — same noisy image, same timestep, only the conditioning changes.

Step 2 — Two arrows from the same starting dot

WHAT. We now draw both arrows from the same noisy image: $ϵ_{c}$ (prompt-aware, magenta) and $ϵ_{\emptyset}$ (prompt-blind, violet). Same start, different tips.

WHY two. A single denoising path can't tell us what the prompt contributed — maybe the arrow points that way for reasons that have nothing to do with your words. The only way to isolate "the prompt's effect" is to run the model a second time with the prompt removed and compare. That comparison is the whole trick.

PICTURE. The two tips land in different places. The gap between the tips is the star of the next step.

Step 3 — The difference arrow: what the prompt actually adds

WHAT. Subtract tip-to-tip. First, formally define the difference arrow: $Δ ϵ := ϵ_{c} - ϵ_{\emptyset}$ (read " $:=$ " as is defined to be; $Δ$ , capital "delta", is the standard symbol for a change or difference). Now the same statement, annotated term by term: $"what the prompt added" Δ ϵ = prompt-aware ϵ_{c} - prompt-blind ϵ_{\emptyset}$

We take the prompt-aware arrow and remove the part it shares with the prompt-blind arrow. What survives is purely the prompt's contribution.

WHY subtract. Vector subtraction $ϵ_{c} - ϵ_{\emptyset}$ draws the arrow from the tip of $ϵ_{\emptyset}$ to the tip of $ϵ_{c}$ . That is literally "the extra push the words gave us." Everything both arrows agreed on cancels; only the disagreement — the "cat-ness" — remains.

PICTURE. The orange arrow is $Δ ϵ$ , floating between the two tips. Notice it does not start at the origin dot; it starts where the violet arrow ends.

Step 4 — Rebuild the arrows: baseline + one hop of the difference

WHAT. Notice a fact that is true by pure algebra: $ϵ_{c} = ϵ_{\emptyset} + Δ ϵ = ϵ_{\emptyset} + 1 \cdot (ϵ_{c} - ϵ_{\emptyset})$

Term by term: start at the blind baseline $ϵ_{\emptyset}$ , take one hop of the difference arrow $Δ ϵ$ , and you arrive exactly at $ϵ_{c}$ . The coefficient in front of $Δ ϵ$ is $1$ .

WHY rewrite something we already knew. Because now the number $1$ is sitting alone, begging to be changed. If one hop lands on "normal cat," what does two hops do? That single question is classifier-free guidance.

PICTURE. The violet baseline, then the orange hop of length one, landing precisely on the magenta tip.

Step 5 — Turn the coefficient into a knob $w$

WHAT. Replace the lonely $1$ with a dial called $w$ , the guidance scale. Keeping every argument in place so you never lose track of what the function depends on: $guided noise, still a function of x_{t}, t, c \tilde{ϵ}_{θ} (x_{t}, t, c) = baseline ϵ_{\emptyset} ϵ_{θ} (x_{t}, t, \emptyset) + volume knob w \cdot prompt direction Δ ϵ (ϵ_{θ} (x_{t}, t, c) - ϵ_{θ} (x_{t}, t, \emptyset))$

Term by term: the tilde $\tilde{ϵ}$ ("epsilon-tilde") means guided noise — the doctored arrow we actually feed back into sampling. It still lives at the same noisy image $x_{t}$ and timestep $t$ ; only its length changed. $w$ multiplies only the difference arrow, so it stretches or shrinks how far past the baseline we hop.

WHY multiply the difference and not the whole thing. If we scaled the whole $ϵ_{c}$ we'd also amplify the generic, prompt-irrelevant part. Scaling only $Δ ϵ$ amplifies exactly the prompt's contribution and nothing else. That's the entire point of splitting the arrows in Steps 2–3.

PICTURE. Same baseline, but now the orange hop is stretched to length $w > 1$ , overshooting the magenta tip toward "extra-cat." This is the "prompt volume knob."

Step 6 — Read every value of the knob (all cases)

WHAT. Slide $w$ across its whole range — including negative values — and watch where the guided tip $\tilde{ϵ}_{θ} (x_{t}, t, c)$ lands. This is the "cover every case" step — no value of $w$ , of either sign, should surprise you.

$w$	weight on $ϵ_{\emptyset}$ is $(1 - w)$	where the tip lands	meaning
$< 0$	$> 1$	on the opposite side of the violet tip from the prompt	anti-prompt / negative guidance — hop backwards along $Δ ϵ$ , pushing away from the prompt (do the opposite of what was asked)
$0$	$+ 1$	on the violet tip	pure blind — prompt ignored
$0 < w < 1$	positive, between $1$ and $0$	between violet and magenta tips	muted / under-guidance — a blend that leans partly toward the prompt but not fully; a weighted interpolation of blind and normal conditional
$1$	$0$	on the magenta tip	normal Conditional generation
$> 1$	negative	past the magenta tip	extrapolation — over-follow the prompt
very large	very negative	far past	over-saturated, low diversity (mode collapse)

WHY the $0 < w < 1$ row exists. For these values both weights are positive and sum to one, so the guided arrow lands on the segment between the two tips — you get a genuine mixture of "generic image" and "prompt image," a softened version of the prompt effect. It never overshoots, because the hop is shorter than the full $Δ ϵ$ .

WHY the $w < 0$ row exists. A negative $w$ multiplies $Δ ϵ$ by a negative number, so the hop runs the other way — starting from the blind baseline and moving directly away from the prompt tip. This is exactly the mechanism behind Negative prompting: you steer generation to avoid a concept by pointing $Δ ϵ$ in reverse. (In practice, the "negative prompt" trick usually swaps $ϵ_{\emptyset}$ for a second prompt's prediction, but the sign idea is the same — subtract that direction rather than add it.)

WHY the negative weight matters (for $w > 1$ ). When $w > 1$ , the coefficient $(1 - w)$ on $ϵ_{\emptyset}$ flips negative. A negative weight pushes away from the generic prediction — the model is told "be less like a random image, more like the thing I asked for." That push-away is why high $w$ sharpens prompt adherence.

PICTURE. One line, dots on it: a $w < 0$ anti-prompt point on the far side, $w = 0$ , a muted $0 < w < 1$ point between the tips, $w = 1$ , and two overshoot points — all lying on the same straight track through the two tips.

Step 7 — The degenerate case: where does $ϵ_{\emptyset}$ come from?

WHAT. Everything above assumed we have a prompt-blind prediction. But a text-to-image model only ever saw prompts! The fix lives in training: on each training example we flip a biased coin, and with probability $p_{uncond}$ we erase the prompt and replace it with the null token $\emptyset$ .

WHAT is $p_{uncond}$ , exactly. It is a single training hyperparameter — a fixed number you choose before training (typically $p_{uncond} \in [0.1, 0.2]$ , i.e. $10 - 20%$ ). Concretely: for every $(x_{0}, c)$ pair drawn in a minibatch, sample a uniform random number $u \in [0, 1]$ ; if $u < p_{uncond}$ , set $c \leftarrow \emptyset$ for that example, otherwise keep the real prompt. Everything else (the loss, the optimizer) stays identical. So $p_{uncond}$ literally sets what fraction of gradient steps see a blank prompt.

WHY that value and not $0$ or $1$ . If $p_{uncond} = 0$ , the model never sees $\emptyset$ , so $ϵ_{\emptyset}$ is undefined and the Step 3 subtraction is meaningless — the degenerate case that breaks guidance. If $p_{uncond} = 1$ , the model never sees prompts and can't do $ϵ_{c}$ at all. A small-but-nonzero value ( $10 - 20%$ ) buys a competent blind prediction while spending most of the budget on the conditional one we care about most.

PICTURE. A training batch: most rows keep the prompt (magenta), a random $p_{uncond}$ fraction have it swapped for $\emptyset$ (violet). Same loss $∥ ϵ - ϵ_{θ} ∥^{2}$ everywhere. Links to DDPM sampling where the two calls are made at inference.

The one-picture summary

WHAT. The entire derivation, compressed: one starting dot, two model arrows, their difference, and the knob that stretches that difference into the guided prediction — with the whole family of $w$ -values (including the negative anti-prompt side) drawn as one track.

Recall Feynman retelling — say it in plain words

Picture a foggy image at some noise level $t$ . Ask the model twice: once knowing what you want ("a cat"), once knowing nothing. Each answer is an arrow pointing "this way to clean up the noise." Lay both arrows from the same spot. Subtract tip-to-tip and you get a little arrow that means "here's the part that is about the cat." Now the trick: start from the clueless arrow, and instead of adding the cat-arrow once, add it several times. You overshoot toward "very cat." The multiplier is $w$ . Turn it to $0$ and you get a random picture; between $0$ and $1$ you get a muted blend; turn it to $1$ and you get an ordinary cat; turn it past $1$ and the cat gets bolder — until, cranked too high, every picture looks the same over-saturated cat. Turn it negative and you get the anti-cat: the hop runs backwards, actively avoiding what you named. And the only reason $ϵ_{\emptyset}$ exists at all is that during training we sometimes hid the prompt (a fixed fraction $p_{uncond}$ of the time) so the model learned to guess blind. That's it — no classifier, just two calls and a knob.

Recall Check yourself

What does $Δ ϵ := ϵ_{c} - ϵ_{\emptyset}$ represent geometrically? ::: The arrow from the tip of the prompt-blind prediction to the tip of the prompt-aware one — the pure contribution of the prompt. What is the timestep $t$ ? ::: Which rung of the noising ladder we are on — large $t$ is very noisy, small $t$ is nearly clean; the model is called once per timestep. Why does $(1 - w)$ go negative for $w > 1$ ? ::: Because the guided arrow overshoots past $ϵ_{c}$ , which means pushing away from the generic $ϵ_{\emptyset}$ , i.e. subtracting it. What happens for $0 < w < 1$ ? ::: A muted (under-guided) blend: the tip lands between the blind and conditional tips, a weighted interpolation of the two. What happens at $w = 0$ and $w = 1$ ? ::: $w = 0$ lands on $ϵ_{\emptyset}$ (prompt ignored); $w = 1$ lands on $ϵ_{c}$ (ordinary conditional). What does $w < 0$ do? ::: Anti-prompt / negative guidance — the hop runs backwards along $Δ ϵ$ , pushing the sample away from the prompt (the sign idea behind negative prompting). Why must we drop the prompt during training, and what sets how often? ::: So a single network learns the blind prediction $ϵ_{\emptyset}$ ; the hyperparameter $p_{uncond}$ (typically 10–20%) fixes the fraction of training examples whose prompt is replaced by $\emptyset$ .

See also: Negative prompting, Prompt engineering, and the Hinglish version.