Generative Models

Question 1

(a) (6) Start with the identity for any $q_\phi$: $\log p_\theta(x)=\mathbb{E}_{q_\phi(z\mid x)}\Big[\log\frac{p_\theta(x,z)}{q_\phi(z\mid x)}\Big]+\mathbb{E}_{q_\phi(z\mid x)}\Big[\log\frac{q_\phi(z\mid x)}{p_\theta(z\mid x)}\Big].$ (2 — inserting $q$ and using $p(x,z)=p(z|x)p(x)$) The first term is the ELBO; the second is $D_{\mathrm{KL}}(q_\phi(z|x)\|p_\theta(z|x))$. (2) Since $\log p_\theta(x)$ is constant in the expectation (does not depend on $z$), the identity holds; KL $\ge0$ by Gibbs/Jensen, so ELBO $\le\log p_\theta(x)$. (1) Maximizing ELBO over $\theta,\phi$ simultaneously (i) tightens the bound on the true log-likelihood and (ii) drives $q_\phi$ toward the true posterior — hence valid surrogate objective. (1)

(b) (6) KL between two $d$-dim Gaussians $\mathcal N(\mu,\Sigma_1)$ and $\mathcal N(0,I)$: $D_{KL}=\tfrac12\Big[\mathrm{tr}(\Sigma_1)+\mu^\top\mu-d-\ln\det\Sigma_1\Big].$ (3 — general Gaussian KL) With $\Sigma_1=\mathrm{diag}(\sigma_j^2)$: $\mathrm{tr}=\sum\sigma_j^2$, $\mu^\top\mu=\sum\mu_j^2$, $\ln\det=\sum\ln\sigma_j^2$, $d=\sum 1$. (2) $\Rightarrow D_{KL}=\tfrac12\sum_j(\mu_j^2+\sigma_j^2-\ln\sigma_j^2-1).\quad\blacksquare$ (1)

(c) (6) Under $z=\mu_\phi+\sigma_\phi\odot\epsilon$ with $\epsilon\sim\mathcal N(0,I)$, the pushforward of $\mathcal N(0,I)$ is exactly $\mathcal N(\mu_\phi,\mathrm{diag}\sigma_\phi^2)=q_\phi$. (2) Therefore $\mathbb{E}_{q_\phi}[f(z)]=\mathbb{E}_{\epsilon}[f(\mu_\phi+\sigma_\phi\odot\epsilon)].$ The RHS distribution $(\mathcal N(0,I))$ is independent of $\phi$, so $\nabla_\phi$ passes inside the expectation (dominated convergence / differentiating under the integral): $\nabla_\phi\mathbb{E}_{q_\phi}[f]=\mathbb{E}_\epsilon[\nabla_\phi f(\mu_\phi+\sigma_\phi\odot\epsilon)].\quad\blacksquare$ (2) Score-function estimator $\nabla_\phi\mathbb E_q[f]=\mathbb E_q[f(z)\nabla_\phi\log q_\phi(z|x)]$ does not use $\nabla f$; it treats $f$ as a black-box weight, so its variance scales with the magnitude of $f$ and does not shrink with a well-behaved gradient — empirically much higher variance. (2)

(d) (4) As $\sigma_j\to0^+$: $\sigma_j^2\to0$ but $-\ln\sigma_j^2\to+\infty$, so the per-dim term $\tfrac12(\mu_j^2+\sigma_j^2-\ln\sigma_j^2-1)\to+\infty$. (2) Thus KL diverges to $+\infty$; the encoder becomes deterministic, ELBO $\to-\infty$, gradients blow up — training penalizes collapsing the posterior to a point. This is the KL regularizer's role: it keeps the latent stochastic and close to the prior. (2)

Question 2

(a) (8) $V=\int p_r(x)\log D(x)+p_g(x)\log(1-D(x))\,dx$. (1) Pointwise maximize $a\log D+b\log(1-D)$: derivative $a/D-b/(1-D)=0\Rightarrow D^*=a/(a+b)=\dfrac{p_r}{p_r+p_g}$. (3) Substitute: $V(D^*)=\mathbb E_{p_r}\Big[\log\frac{p_r}{p_r+p_g}\Big]+\mathbb E_{p_g}\Big[\log\frac{p_g}{p_r+p_g}\Big].$ (2) Write $p_r+p_g=2m$ where $m=\tfrac{p_r+p_g}{2}$: $=-2\log2+D_{KL}(p_r\|m)+D_{KL}(p_g\|m)=2D_{JS}(p_r\|p_g)-2\log2.\quad\blacksquare$ (2)

(b) (4) When supports are disjoint, $D^*$ can be made $1$ on real, $0$ on fake exactly ⇒ $D_{JS}=\log2$, so $V(D^*)=0$ constant, and generator gradient $\nabla_G$ through $\log(1-D)$ vanishes (saturation). (2) No useful signal reaches $G$; it cannot spread mass, and any narrow region that fools $D$ is reinforced → generator maps to few modes = mode collapse. (2)

(c) (5) For $p_r=\delta_0$, $p_g=\delta_\theta$: EM distance is $|θ|$ (mass moved distance $|\theta|$): $W=|\theta|$. (2) JS: for $\theta\ne0$ supports disjoint so $D_{JS}=\log2$ (constant); at $\theta=0$, $D_{JS}=0$. (2) $W(\theta)=|\theta|$ is continuous with a.e. nonzero gradient $\mathrm{sign}(\theta)$ pointing toward $\theta=0$, while $D_{JS}$ is flat (zero gradient) for all $\theta\ne0$. Hence WGAN provides usable gradients everywhere; standard GAN does not. (1)

(d) (3) The $\|f\|_L\le1$ Lipschitz constraint is required for the Kantorovich–Rubinstein duality to equal $W$; without it the sup is unbounded. (1.5) Enforcement: weight clipping (original WGAN) or gradient penalty (WGAN-GP) / spectral normalization. (1.5)

Question 3

(a) (6) Base: $x_1=\sqrt{\alpha_1}x_0+\sqrt{1-\alpha_1}\epsilon_1$, matches $\bar\alpha_1=\alpha_1$. (1) Inductive step: assume $x_{t-1}=\sqrt{\bar\alpha_{t-1}}x_0+\sqrt{1-\bar\alpha_{t-1}}\,\bar\epsilon_{t-1}$. Then $x_t=\sqrt{\alpha_t}x_{t-1}+\sqrt{1-\alpha_t}\epsilon_t=\sqrt{\alpha_t\bar\alpha_{t-1}}x_0+\big(\sqrt{\alpha_t(1-\bar\alpha_{t-1})}\,\bar\epsilon_{t-1}+\sqrt{1-\alpha_t}\,\epsilon_t\big).$ (2) The two independent Gaussians sum in variance: $\alpha_t(1-\bar\alpha_{t-1})+(1-\alpha_t)=1-\alpha_t\bar\alpha_{t-1}=1-\bar\alpha_t$. (2) Mean coefficient $=\sqrt{\bar\alpha_t}$. Hence $q(x_t|x_0)=\mathcal N(\sqrt{\bar\alpha_t}x_0,(1-\bar\alpha_t)I)$. $\blacksquare$ (1)

(b) (6) $\epsilon_\theta(x_t,t)$ predicts the noise $\epsilon$ that was added to produce $x_t$ from $x_0$. (2) Since $x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon$, we have $\nabla_{x_t}\log q(x_t|x_0)=-\dfrac{x_t-\sqrt{\bar\alpha_t}x_0}{1-\bar\alpha_t}=-\dfrac{\epsilon}{\sqrt{1-\bar\alpha_t}}$. (2) Thus the score is a rescaling of $-\epsilon$; predicting $\epsilon$ is equivalent to score matching with $s_\theta=-\epsilon_\theta/\sqrt{1-\bar\alpha_t}$, differing only by the $t$-dependent weight that $L_{\text{simple}}$ drops. (2)

(c) (6) $\bar\alpha_t=\prod(1-\beta_s)$ decreases monotonically from $\bar\alpha_0=1$ toward ≈0. Early $\beta$ small ⇒ $\bar\alpha_t$ near 1 (little noise); late steps drive it to ~0. (2) $\bar\alpha_T\approx0$ makes $q(x_T|x_0)\approx\mathcal N(0,I)$ independent of $x_0$, so sampling can begin from pure Gaussian noise (matching the prior). (2) Toy schedule $\beta=(0.1,0.2,0.3)$: $\alpha=(0.9,0.8,0.7)$. $\bar\alpha_1=0.9$, $\bar\alpha_2=0.72$, $\bar\alpha_3=0.504$. (2)

[
{"claim":"Diagonal-Gaussian KL to N(0,I) formula for mu=[1,2], sigma2=[0.5,1.5]",
 "code":"import sympy as sp\nmu=[sp.Integer(1),sp.