Exercises — Classifier-free guidance

• ${ϵ}_{\theta }({\mathbf{x}}_t,t,\varnothing )$ = the noise the model predicts when it is told nothing (empty prompt). One network call.
• ${ϵ}_{\theta }({\mathbf{x}}_t,t,\mathbf{c})$ = the noise predicted when it sees the prompt $\mathbf{c}$. One network call.
• $\Delta ϵ={ϵ}_{\theta }({\mathbf{x}}_t,t,\mathbf{c})-{ϵ}_{\theta }({\mathbf{x}}_t,t,\varnothing )$ = the direction the prompt adds. Because it is a difference, computing it needs both calls — so two network evaluations per denoising step.

Plug into the linear form ${\widetilde{ϵ}}_{\theta }=(1-w){ϵ}_{\varnothing }+w{ϵ}_{\mathbf{c}}$.

• $w=1$: $\widetilde{ϵ}=0\cdot {ϵ}_{\varnothing }+1\cdot {ϵ}_{\mathbf{c}}={ϵ}_{\mathbf{c}}$ → ordinary conditional.
• $w=0$: $\widetilde{ϵ}=1\cdot {ϵ}_{\varnothing }+0={ϵ}_{\varnothing }$ → purely unconditional.

Step 1 — direction. Subtract component-by-component: $\Delta ϵ=[0.4-0.1,-0.2-0.1,0.6-0.2]=[0.3,-0.3,0.4].$ Step 2 — scale & add. ${\widetilde{ϵ}}_{\theta }={ϵ}_{\varnothing }+w\Delta ϵ=[0.1,0.1,0.2]+3.0[0.3,-0.3,0.4].$ $=[0.1+0.9,0.1-0.9,0.2+1.2]=[1.0,-0.8,1.4].$

With $w=3$: $(1-3){ϵ}_{\varnothing }+3{ϵ}_{\mathbf{c}}=-2{ϵ}_{\varnothing }+3{ϵ}_{\mathbf{c}}$. $-2[0.1,0.1,0.2]+3[0.4,-0.2,0.6]=[-0.2,-0.2,-0.4]+[1.2,-0.6,1.8]=[1.0,-0.8,1.4].✓$ Why identical: the two forms are algebraically the same expression — ${ϵ}_{\varnothing }+w({ϵ}_{\mathbf{c}}-{ϵ}_{\varnothing })={ϵ}_{\varnothing }+w{ϵ}_{\mathbf{c}}-w{ϵ}_{\varnothing }=(1-w){ϵ}_{\varnothing }+w{ϵ}_{\mathbf{c}}.$ Just factoring — nothing physical changes.

Look at the second figure. Place ${ϵ}_{\varnothing }$ and ${ϵ}_{\mathbf{c}}$ as two points. The guided output lies on the line through them, at parameter $w$: $\widetilde{ϵ}(w)={ϵ}_{\varnothing }+w({ϵ}_{\mathbf{c}}-{ϵ}_{\varnothing }).$

• $w\in [0,1]$: $\widetilde{ϵ}$ sits between the two points (interpolation).
• $w>1$: $\widetilde{ϵ}$ shoots past ${ϵ}_{\mathbf{c}}$, on the far side, moving away from ${ϵ}_{\varnothing }$ (extrapolation). The coefficient sum is always $(1-w)+w=1$, so this is an affine combination — the point stays on the line. A negative weight on ${ϵ}_{\varnothing }$ literally means "step in the direction opposite the generic guess," i.e. push away from bland/unconditional content toward the prompt.

$(1-w)+w=1$ for every $w$. This is why $\widetilde{ϵ}$ stays on the line through the two predictions. If the coefficients summed to something else, say $k\ne 1$, the guided noise would be scaled by roughly $k$ overall. In the DDPM sampling update, an over-scaled noise estimate would systematically over- or under-remove noise each step, drifting ${\mathbf{x}}_t$ off the data manifold — you'd get blown-out or washed-out images independent of the prompt. The unit sum keeps the magnitude honest and lets $w$ control only direction.

Bayes gives $\log p({\mathbf{x}}_t\mid \mathbf{c})=\log p({\mathbf{x}}_t)+\log p(\mathbf{c}\mid {\mathbf{x}}_t)-\log p(\mathbf{c})$. Gradient in ${\mathbf{x}}_t$ kills $\log p(\mathbf{c})$: $\nabla \log p(\mathbf{c}\mid {\mathbf{x}}_t)=\nabla \log p({\mathbf{x}}_t\mid \mathbf{c})-\nabla \log p({\mathbf{x}}_t).$ Substitute into CFG's guided score: $\nabla \log \tilde{p}=\nabla \log p({\mathbf{x}}_t)+w\nabla \log p(\mathbf{c}\mid {\mathbf{x}}_t).$ This is exactly the Classifier guidance form (unconditional score + $w\times$ classifier gradient) — but the "classifier gradient" is not from a separately trained classifier; it is the implicit classifier $\nabla \log p(\mathbf{c}\mid {\mathbf{x}}_t)=\nabla \log p({\mathbf{x}}_t\mid \mathbf{c})-\nabla \log p({\mathbf{x}}_t),$ obtained purely from the two noise predictions the single model already gives. That is the whole point: no extra classifier.

Swap $\varnothing \to {\mathbf{c}}_{\text{neg}}$ in the additive form: $\widetilde{ϵ}={ϵ}_{\theta }({\mathbf{x}}_t,t,{\mathbf{c}}_{\text{neg}})+w[{ϵ}_{\theta }({\mathbf{x}}_t,t,\mathbf{c})-{ϵ}_{\theta }({\mathbf{x}}_t,t,{\mathbf{c}}_{\text{neg}})].$ Geometrically, the baseline point moves from "generic" to "the thing you hate," and $\Delta ϵ$ now points from the disliked image toward the wanted one. Extrapolating along it ($w>1$) actively repels the sample from "blurry/deformed" while pulling toward $\mathbf{c}$. Ordinary CFG is the special case ${\mathbf{c}}_{\text{neg}}=\varnothing$.

Count. Unguided: $50\times 1=50$ passes. Guided (naive CFG): $50\times 2=100$ passes — exactly double. Cost vs $w$. $w$ is just a scalar multiplying an already-computed $\Delta ϵ$; changing it is a cheap vector scale, no new forward pass. So compute is flat in $w$. Why quality still degrades. As $w$ grows, $\widetilde{ϵ}$ extrapolates far past ${ϵ}_{\mathbf{c}}$. In Score-based diffusion models this over-sharpens the score, concentrating probability mass onto a few high-likelihood modes → reduced diversity and, at extreme $w$ (≈15), over-saturated, unnatural colours (the parent's Example 3). Large $w$ also amplifies the magnitude of $\widetilde{ϵ}$, over-removing noise per step and pushing ${\mathbf{x}}_t$ off the manifold. The sweet spot is empirical: $w\in [7,10]$ for Stable-Diffusion-style text-to-image.

Batch trick. Stack the conditional and unconditional inputs into one batch of size 2: feed $[{\mathbf{x}}_t;{\mathbf{x}}_t]$ with conditions $[\mathbf{c};\varnothing ]$. One batched forward call returns both ${ϵ}_{\mathbf{c}}$ and ${ϵ}_{\varnothing }$ simultaneously. This is still "two evaluations of work" but a single kernel launch / single call, so wall-clock ≈ one large pass instead of two sequential ones. Tradeoff: memory doubles (batch ×2) but latency drops from $2\times$ sequential to ≈$1\times$ (given hardware parallelism). If memory is the bottleneck you cannot use it and must pay the sequential $2\times$ time. Net: trade GPU memory for speed; you cannot get CFG for the math of one pass because $\Delta ϵ$ intrinsically needs both predictions.

CFG cost per step ::: two noise predictions ($\mathbf{c}$ and $\varnothing$), combined as an affine mix whose coefficients always sum to 1, with $w$ controlling extrapolation strength.