Classifier-free guidance

What Problem Does It Solve?

Conditional diffusion models (like text-to-image systems) often produce samples that are high-quality but don't strongly follow the conditioning signal (text, class label, etc.). Early approaches used a separate classifier gradient to steer generation, but this required training an extra noise-robust classifier.

Classifier-free guidance eliminates the classifier entirely by training a single model to handle both conditional and unconditional generation, then combining their predictions at sampling time.

The Mathematical Framework

Derivation from First Principles

STEP 1: What are we trying to achieve?

We want to sample from $p(\mathbf{x} | \mathbf{c})$ but make it more strongly reflect the conditioning $\mathbf{c}$. In classifier guidance, we modified the score with $\nabla_{\mathbf{x}_t} \log p(\mathbf{c} | \mathbf{x}_t)$. Here we'll derive an equivalent effect without a classifier.

STEP 2: Rewrite the conditional score

By Bayes' rule: $\log p(\mathbf{x}_t | \mathbf{c}) = \log p(\mathbf{x}_t) + \log p(\mathbf{c} | \mathbf{x}_t) - \log p(\mathbf{c})$

Taking gradients: $\nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t | \mathbf{c}) = \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t) + \nabla_{\mathbf{x}_t} \log p(\mathbf{c} | \mathbf{x}_t)$

WHY this step? The last term $\log p(\mathbf{c})$ doesn't depend on $\mathbf{x}_t$, so it vanishes under the gradient. This splits the conditional score into unconditional score + classifier gradient.

STEP 3: Amplify the conditioning signal

To make the sample follow $\mathbf{c}$ more strongly, we scale up the classifier term by a factor $w$: $\nabla_{\mathbf{x}_t} \log \tilde{p}(\mathbf{x}_t | \mathbf{c}) = \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t) + w \cdot \nabla_{\mathbf{x}_t} \log p(\mathbf{c} | \mathbf{x}_t)$

WHY? When $w = 1$, we get the true conditional. When $w > 1$, we "overshoot" toward the conditioning, making outputs more specific to $\mathbf{c}$ at the cost of diversity.

STEP 4: Express in terms of the model's predictions

Rearrange the Bayes gradient: $\nabla_{\mathbf{x}_t} \log p(\mathbf{c} | \mathbf{x}_t) = \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t | \mathbf{c}) - \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t)$

Substitute into the guided score: $\nabla_{\mathbf{x}_t} \log \tilde{p}(\mathbf{x}_t | \mathbf{c}) = \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t) + w \cdot [\nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t | \mathbf{c}) - \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t)]$

$= (1 - w) \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t) + w \, \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t | \mathbf{c})$

STEP 5: Convert to noise prediction

In diffusion models, the score relates to predicted noise: $\nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t) \propto -\epsilon_\theta(\mathbf{x}_t, t)$. So:

$\boxed{\tilde{\epsilon}_\theta = (1 - w) \epsilon_\theta(\mathbf{x}_t, t, \emptyset) + w \, \epsilon_\theta(\mathbf{x}_t, t, \mathbf{c})}$

WHY this form? It's a linear interpolation and extrapolation. When $w = 0$, purely unconditional. When $w = 1$, normal conditional. When $w > 1$, we extrapolate beyond the conditional toward "more $\mathbf{c}$."

Training Procedure

To enable classifier-free guidance, we train a single model that handles both conditional and unconditional denoising:

1. During training, randomly drop the conditioning $\mathbf{c}$ with probability $p_{\text{uncond}}$ (typically 10-20%).

   • When dropped, replace $\mathbf{c}$ with a null token $\emptyset$ (empty string for text, zero embedding, etc.).

2. Loss remains the same: $\mathcal{L} = \mathbb{E}_{t, \mathbf{x}_0, \epsilon, \mathbf{c}} \left[ \| \epsilon - \epsilon_\theta(\mathbf{x}_t, t, \mathbf{c}) \|^2 \right]$ where $\mathbf{c}$ is sometimes $\emptyset$.

WHY this works: The model learns to denoise both with and without conditioning in the same parameters. At inference, we can call it twice (once with $\mathbf{c}$, once with $\emptyset$) to get both predictions.

Worked Examples

Common Mistakes

Connections to Other Concepts

• Score-based diffusion models: CFG manipulates the score function by combining conditional and unconditional scores.
• Classifier guidance: The predecessor approach that used a separate classifier $p(\mathbf{c} | \mathbf{x}_t)$; CFG derives the same effect without classifier.
• DDPM sampling: CFG modifies the noise prediction $\epsilon_\theta$ used in the DDPM reverse process.
• Conditional generation: CFG is a technique to strengthen conditioning signals in conditional generative models.
• Prompt engineering: Higher guidance scales make prompt wording more critical (overfitting to text).
• Negative prompting: Often combined with CFG to push away from unwanted features.

When to Use CFG

Use classifier-free guidance when:

• You want strong adherence to text/class conditioning without training a separate classifier
• You're building text-to-image, text-to-video, or class-conditional models
• You need a tunable knob for prompt strength

Avoid or tune down when:

• You want maximum sample diversity (high $w$ causes mode collapse)
• Compute cost is critical (requires 2× model evaluations per step)
• You're generating unconditional samples (just use $w = 0$ or skip CFG)

#flashcards/ai-ml

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)