4.5.13 · D5Generative Models
Question bank — Score-based generative models
Two symbols recur below, so let's pin them once in plain words:
- The score is an arrow at every point telling you which direction increases probability fastest, and how steeply. It has one arrow-component per data dimension.
- The partition function is the single number that rescales an unnormalized "shape" so the total probability equals . It is one number, not a function of .
True or false — justify
The score is a scalar number at each point
False — it is a vector, one gradient-component per data dimension; it points toward higher probability, so it carries direction, not just magnitude.
Learning the score lets you recover the exact density for free
Not for free — the score is , so recovering needs integrating the score field and re-inserting the normalizer , which is exactly the hard quantity score methods were designed to avoid.
Because drops out of the score, the model never has to care about normalization at all
True for training and sampling — since is constant in — but if you ever want a likelihood number, comes back and must be handled separately.
Denoising score matching trains the network to reconstruct the clean image
False — it trains the network to predict , the direction back toward clean data, i.e. the score of the noised distribution, not the clean pixels themselves.
A single well-chosen noise level is enough to model any dataset
False — one either blurs fine structure (if large) or leaves far-from-data regions with no learned signal (if small); you need a ladder of 's to cover both.
The weighting changes which function is optimal
False — at the true minimum every term is zero regardless of positive weights; only rebalances gradient magnitudes across scales so no level dominates training.
Langevin dynamics without the noise term still samples correctly
False — dropping the noise turns it into plain gradient ascent, which collapses onto modes (peaks) instead of sampling the full distribution; the injected noise is what gives you a distribution, not a maximizer.
Annealed Langevin and DDPM sampling are unrelated algorithms
False — both descend a score field from noise to data; DDPM's -prediction is a score up to the scaling, so they are two views of the same idea.
The score always points exactly at the nearest data point
False — it points up the log-density gradient, which is a smoothed blend of all nearby data; near a mode it points inward, but between two modes it can point along a ridge, not at any single sample.
Predicting and predicting the score are genuinely different training targets
False — since the true score of is , the two targets differ only by the known factor ; the network learns the same information.
Spot the error
"Since depends on the data, ."
Wrong — integrates out, leaving a constant that does not vary with the input ; therefore . The confusion is between the integration variable and the input.
"The naïve score-matching loss is directly usable because we can just plug in the data."
Wrong — we never have ; it is the unknown we want. That's precisely why denoising (or implicit) score matching exists — to sidestep the unavailable target.
"For Gaussian noise , the score is ."
Missing the variance: it is . Forgetting the makes small-noise scores explode incorrectly and breaks the weighting logic.
"In annealed Langevin we should use the same step size at every noise level for stability."
Wrong — small needs a smaller step (finer landscape, sharper score), so should shrink with (commonly ); a fixed large step overshoots at fine scales.
"Because the score removes , score-based models can compute exact log-likelihoods trivially."
Wrong — removing helps training and sampling, not likelihood evaluation; exact likelihoods require solving the probability-flow ODE or reintroducing normalization, both nontrivial.
"Start Langevin sampling from the data to reach the data faster."
Wrong — you must start from the high-noise prior that the largest- score was trained on; starting elsewhere lands you where the network never learned a valid score.
Why questions
Why the gradient of the log probability rather than the probability itself?
The converts into , so the constant separates and its gradient vanishes — this is exactly what makes disappear and the score tractable.
Why add noise at all, instead of matching the score of the raw data?
Real data sits on a thin manifold where is undefined or huge off-manifold; noise smears the distribution into all of space so a smooth, learnable score exists everywhere.
Why does the term have a square root, not just ?
It's a standard deviation of injected Gaussian noise; the Wiener increment's variance scales like the step , so its magnitude (std) scales like , giving .
Why anneal from large down to small rather than the reverse?
Large gives a smooth, connected landscape so the chain can travel between modes; shrinking then sharpens detail. Starting fine would trap the chain in whatever region it began.
Why weight the multi-scale loss by ?
The target has magnitude , so its squared error scales like ; multiplying by makes each level contribute comparably and prevents small- terms from dominating gradients.
Why is the score often easier to learn than the density in high dimensions?
The density fights the intractable integral over the whole space, whereas the score is a local quantity estimable from noisy-clean pairs — no global normalization ever needed.
Why does predicting end up teaching the network the true score?
Because the analytic score of the Gaussian-noised conditional is exactly , so minimizing to that target drives toward the true (marginal) score at that noise level.
Edge cases
What is the score of a uniform distribution on its support?
Zero everywhere inside — is constant, so its gradient vanishes; there is no "uphill" direction, and Langevin reduces to a pure random walk within the support.
What happens to the score as near the data manifold?
The score blows up in magnitude, and off-manifold points get an enormous restoring pull; this is why the smallest needs tiny step sizes and why one can't train at .
What happens as (very large noise)?
The noised distribution flattens toward a broad Gaussian centered at the data mean; the score becomes gentle and nearly linear, giving the well-connected landscape that lets the chain explore freely.
At a probability peak (mode), what does the score equal?
Exactly zero — the gradient of vanishes at a maximum; the deterministic drift stops there, and only the injected noise keeps the sample from freezing on the peak.
Between two equal, well-separated modes, what does the score do?
It points away from the low-density valley toward whichever mode you're nearer; at the exact saddle it can be zero or tangential, so crossing between modes relies on the high- smoothing plus noise.
If two data points coincide (a duplicated sample), does the score break?
No — duplication just doubles that point's contribution to the smoothed density; the score stays finite and simply pulls a bit harder toward that location.
What if you set the number of Langevin steps per level to 1?
The chain barely relaxes toward the current noise level's distribution before changes, so samples lag behind the target and quality degrades; enough steps per level are needed to approximately equilibrate.