Generative Models

Question 1 (12 marks)

(a) Start with marginal, multiply/divide by $q_\phi$: (6) $\log p_\theta(x) = \log \int p_\theta(x,z)\,dz = \log \mathbb{E}_{q_\phi(z|x)}\left[\frac{p_\theta(x,z)}{q_\phi(z|x)}\right]$ By Jensen's inequality (log is concave): (1) $\geq \mathbb{E}_{q_\phi}\left[\log \frac{p_\theta(x,z)}{q_\phi(z|x)}\right] = \text{ELBO}. \quad (1)$ Exact identity: write $p_\theta(x,z)=p_\theta(z|x)p_\theta(x)$: (2) $\log p_\theta(x) = \mathbb{E}_{q_\phi}\!\left[\log\frac{p_\theta(x,z)}{q_\phi(z|x)}\right] + \mathbb{E}_{q_\phi}\!\left[\log\frac{q_\phi(z|x)}{p_\theta(z|x)}\right]$ $= \text{ELBO} + D_{KL}(q_\phi(z|x)\|p_\theta(z|x)). \quad (2)$

(b) Since $D_{KL}\geq 0$, ELBO $\leq \log p_\theta(x)$ always (1). Maximizing ELBO pushes the lower bound up; the true posterior $p_\theta(z|x)$ is intractable so we can't optimize likelihood directly (1). The gap equals the KL between approximate and true posterior — tightness of the bound; zero gap means $q_\phi$ matches the true posterior exactly (1).

(c) (3) $\text{ELBO} = \underbrace{\mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)]}_{\text{reconstruction (1)}} - \underbrace{D_{KL}(q_\phi(z|x)\|p(z))}_{\text{regularization / prior-matching (1)}}$ Reconstruction rewards accurate decoding; KL keeps latent code close to prior (1).

Question 2 (10 marks)

(a) $z = \mu_\phi(x) + \sigma_\phi(x)\odot\epsilon$, where $\epsilon\sim\mathcal{N}(0,I)$. (3)

(b) The expectation is over $q_\phi$ which depends on $\phi$; the sampling operation is not differentiable w.r.t. $\phi$ (2). The score-function/REINFORCE estimator works but has high variance (1). Reparameterization moves stochasticity to $\epsilon$ (independent of $\phi$), making $z$ a deterministic differentiable function of $\phi$, so $\nabla_\phi$ passes through and gives a low-variance pathwise estimator (1).

(c) Single-dim KL: (3) $D_{KL} = \int \mathcal{N}(\mu,\sigma^2)\log\frac{\mathcal{N}(\mu,\sigma^2)}{\mathcal{N}(0,1)}$ $= -\tfrac12\left(1 + \log\sigma^2 - \mu^2 - \sigma^2\right) = \tfrac12(\mu^2 + \sigma^2 - \log\sigma^2 - 1).$ (Steps: expand log-ratio, take Gaussian moments $\mathbb{E}[z]=\mu$, $\mathbb{E}[z^2]=\mu^2+\sigma^2$.)

Question 3 (12 marks)

(a) For fixed $G$, maximize integrand pointwise: (5) $V = \int_x \left[p_{data}(x)\log D(x) + p_g(x)\log(1-D(x))\right]dx.$ For $a\log D + b\log(1-D)$, set derivative $\frac{a}{D}-\frac{b}{1-D}=0 \Rightarrow D^* = \frac{a}{a+b}$. So $D^*(x) = \frac{p_{data}(x)}{p_{data}(x)+p_g(x)}. \quad (5)$

(b) Substitute: (5) $V(D^*,G) = \mathbb{E}_{p_{data}}\!\left[\log\frac{p_{data}}{p_{data}+p_g}\right] + \mathbb{E}_{p_g}\!\left[\log\frac{p_g}{p_{data}+p_g}\right].$ Add/subtract $\log 2$ in each: $= -\log 4 + D_{KL}\!\left(p_{data}\Big\|\tfrac{p_{data}+p_g}{2}\right) + D_{KL}\!\left(p_g\Big\|\tfrac{p_{data}+p_g}{2}\right)$ $= -\log 4 + 2\,D_{JS}(p_{data}\|p_g).$ Global minimum when $p_g = p_{data}$, giving $V = -\log 4$.

(c) When supports don't overlap, JS is constant $\log 2$ → gradient vanishes, generator gets no signal → mode collapse (generator maps to few modes to fool $D$) (1). Fix: WGAN uses Earth-Mover distance (with weight clipping / gradient penalty) giving smooth, non-vanishing gradients everywhere (1).

Question 4 (10 marks)

(a) Let $\alpha_t = 1-\beta_t$. Recursively: (6) $x_t = \sqrt{\alpha_t}\,x_{t-1} + \sqrt{1-\alpha_t}\,\epsilon_{t-1}$ Substitute $x_{t-1}=\sqrt{\alpha_{t-1}}x_{t-2}+\sqrt{1-\alpha_{t-1}}\epsilon_{t-2}$: $x_t = \sqrt{\alpha_t\alpha_{t-1}}\,x_{t-2} + \sqrt{\alpha_t(1-\alpha_{t-1})}\epsilon_{t-2} + \sqrt{1-\alpha_t}\epsilon_{t-1}.$ Sum of two independent Gaussians → variance adds: $\alpha_t(1-\alpha_{t-1}) + (1-\alpha_t) = 1-\alpha_t\alpha_{t-1}$ (2). Repeating: $q(x_t\mid x_0) = \mathcal{N}\!\left(x_t; \sqrt{\bar\alpha_t}\,x_0,\ (1-\bar\alpha_t)I\right),\quad \bar\alpha_t=\prod_{s=1}^t \alpha_s. \quad (2)$

(b) $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\quad \epsilon\sim\mathcal{N}(0,I)$. (2)

(c) As $T\to\infty$, $\bar\alpha_T\to 0$ so $q(x_T\mid x_0)\to\mathcal{N}(0,I)$ — independent of $x_0$ (1). This lets sampling begin from pure Gaussian noise then run the learned reverse process (1).

Question 5 (8 marks)

import torch.nn.functional as F
 
def vae_loss(x, x_recon, mu, logvar):
    # reconstruction term: -E[log p(x|z)] via BCE, summed over pixels
    recon = F.binary_cross_entropy(x_recon, x, reduction='sum')
    # KL(N(mu,sigma^2)||N(0,1)) = 0.5*sum(mu^2 + sigma^2 - logvar - 1)
    kl = 0.5 * torch.sum(mu.pow(2) + logvar.exp() - logvar - 1)
    # negative ELBO = reconstruction + KL regularizer
    return recon + kl

Marks: correct BCE recon (2), correct KL formula matching Q2c (3), sign/sum correct (2), comments (1).

Question 6 (8 marks)

(a) FID: (4) $\text{FID} = \|\mu_r - \mu_g\|^2 + \text{Tr}\!\left(\Sigma_r + \Sigma_g - 2(\Sigma_r\Sigma_g)^{1/2}\right).$ Computed on Inception activations. First term = squared distance between mean feature vectors (measures difference in average content) (1); second term = covariance mismatch (measures difference in feature diversity/spread) (1). It is the Fréchet (Wasserstein-2) distance between two Gaussians fit to real/generated features (1). Lower = generated distribution closer to real → better quality & diversity (1).

(b) Guided prediction: (4) $\tilde\epsilon_\theta(x_t,c) = \epsilon_\theta(x_t,\varnothing) + w\big(\epsilon_\theta(x_t,c) - \epsilon_\theta(x_t,\varnothing)\big).$ $\epsilon_\theta(x_t,\varnothing)$ = unconditional prediction (condition dropped during training) (1); the difference is the "conditional direction" (1). $w=0$ → unconditional; $w=1$ → standard conditional; $w>1$ amplifies conditioning → sharper prompt adherence but reduced diversity / possible artifacts (2).

[
  {"claim": "KL(N(mu,sigma^2)||N(0,1)) = 0.5*(mu^2+sigma^2-log(sigma^2)-1)",
   "code": "mu,s=symbols('mu s',positive=True); z=symbols('z'); import sympy as sp; p=1/sp.sqrt(2*pi*s**2)*sp.exp(-(z-mu)**2/(2*s**2)); q=1/sp.sqrt(2*pi)*sp.exp(-z**2/2); kl=sp.integrate(p*sp.log(p/q),(z,-oo,oo)); target=sp.Rational(1,2)*(mu**2+s**2-sp.log(s**2)-1); result=sp.simplify(kl-target)==0"},
  {"claim": "Optimal GAN value equals -log4 + 2*JSD at p_g=p_data i.e. min value = -log(4)",
   "code": "a=symbols('a',positive=True); Dstar=a/(a+a); val=a*sp.log(Dstar)+a*sp.log(1-Dstar); result=sp.simplify(2*val - (-sp.log(4)))==0"},
  {"claim": "DDPM variance recursion: alpha_t*(1-alpha_{t-1})+(1-alpha_t) = 1-alpha_t*alpha_{t-1}",
   "code": "at,a1=symbols('at a1'); lhs=at*(1-a1)+(1-at); rhs=1-at*a1; result=sp.simplify(lhs-rhs)==0"},
  {"claim": "Classifier-free guidance at w=1 reduces to conditional prediction",
   "code": "e_c,e_u,w=symbols('e_c e_u w'); g=e_u+w*(e_c-e_u); result=sp.simplify(g.subs(w,1)-e_c)==0"}
]