Generative Models

Question 1 (14 marks)

(a) (3) $\text{ELBO} = \underbrace{\mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)]}_{\text{reconstruction}} - \underbrace{D_{KL}(q_\phi(z|x)\,\|\,p(z))}_{\text{regularizer}}.$

• Reconstruction term (1): encourages decoded samples to match input $x$.
• KL term (1): pulls the posterior toward the prior, regularizing/structuring the latent space (1).

(b) (4) $\mu^2 = 4.0$ (1); $\sigma^2 = 0.25$ (1); $\ln\sigma^2 = \ln 0.25 = -1.386294$ (1). $D_{KL} = \tfrac12(4.0 + 0.25 - 1 - (-1.386294)) = \tfrac12(4.636294) = 2.318147.$ (1)

(c) (4)

• Sampling z ~ N(mu, sigma²) directly is a stochastic node; gradients cannot flow back through a random draw to $\mu,\sigma$ (1).
• Reparameterization moves randomness to an external variable $\epsilon\sim\mathcal N(0,1)$ (1), making $z = \mu + \sigma\epsilon$ a deterministic, differentiable function of parameters (1).
• Without it, backprop to the encoder parameters is impossible / gradient is high-variance (score-function) and training stalls (1).

(d) (3)

• Larger $\beta$ weights the KL more, forcing posterior closer to prior → more disentangled / smoother latent space (1).
• Reconstruction quality degrades because capacity to encode information is penalized (1); risk of posterior collapse where latents become uninformative (1).

Question 2 (13 marks)

(a) (4) $\alpha_1 = 0.9,\ \alpha_2 = 0.8,\ \alpha_3 = 0.5$ (2). $\bar\alpha_3 = 0.9\times0.8\times0.5 = 0.36.$ (2)

(b) (4) $\sqrt{\bar\alpha_3} = \sqrt{0.36} = 0.6$ (1); $\sqrt{1-\bar\alpha_3} = \sqrt{0.64} = 0.8$ (1). $x_3 = 0.6\times1.0 + 0.8\times0.5 = 0.6 + 0.4 = 1.0.$ (2)

(c) (2) $\bar\alpha_t \to 0$ (1), so $q(x_T|x_0)\to\mathcal N(0,I)$; this lets sampling start from pure Gaussian noise independent of $x_0$ (1).

(d) (3)

• Cosine schedule adds noise more gradually, avoiding the abrupt information destruction of linear schedules near the end (1).
• Keeps more signal in later steps / better distributes difficulty across timesteps (1) → improved sample quality especially at high resolution (1).

Question 3 (12 marks)

(a) (5) For fixed $G$, maximize over $D$ the integrand: $\int p_{data}(x)\log D(x) + p_g(x)\log(1-D(x))\,dx.$ (1) Pointwise, maximize $f(D) = a\log D + b\log(1-D)$ with $a=p_{data},\ b=p_g$ (1). $f'(D) = \frac{a}{D} - \frac{b}{1-D} = 0$ (1) $\Rightarrow a(1-D) = bD \Rightarrow D = \frac{a}{a+b}$ (1). $D^*(x) = \frac{p_{data}(x)}{p_{data}(x)+p_g(x)}.$ (1)

(b) (3) $p_g = p_{data} \Rightarrow D^* = \tfrac12$ (1). Objective value $= \log\tfrac12 + \log\tfrac12 = -2\ln 2 = -1.386294$ (i.e. $\log\frac14$) (2).

(c) (4)

• Generator maps many/all $z$ to a few outputs, ignoring diversity (1).
• Standard loss only rewards fooling $D$; producing one convincing mode still lowers loss, so limited-diversity outputs aren't penalized (1).
• Remedy e.g. WGAN / Wasserstein loss (1): provides smoother gradients and a distance that reflects distribution overlap, reducing collapse (or minibatch discrimination) (1).

Question 4 (11 marks)

(a) (4) $w=1$: $\hat\epsilon = 0.20 + 1(0.50-0.20) = 0.50$ (2). $w=3$: $\hat\epsilon = 0.20 + 3(0.30) = 0.20 + 0.90 = 1.10$ (2).

(b) (3)

• $w=0$: purely unconditional prediction $\epsilon_\theta(x_t,\varnothing)$ — ignores the prompt (1.5).
• $w=1$: standard conditional prediction $\epsilon_\theta(x_t,c)$ (no extra guidance) (1.5).

(c) (4)

• Larger $w$ pushes samples toward stronger alignment with $c$ → better prompt fidelity/sharper class features (2).
• But reduces diversity and can cause over-saturation/artifacts and mode narrowing, since predictions are extrapolated beyond the model's learned distribution (2).

Question 5 (10 marks)

(a) (6)

• GAN: high quality, fast single-pass inference, but unstable training / mode collapse (2).
• Pixel diffusion: excellent quality & stable training, but very expensive inference (many denoising steps at full resolution) (2).
• Latent diffusion: diffuses in a compressed latent space → near-diffusion quality, stable, and much cheaper inference. Recommend latent diffusion given the small GPU budget and text-to-image need (2).

(b) (4)

• FID: Fréchet distance between Gaussian-fit features (Inception activations) of real vs generated images; captures quality and diversity; lower is better (2).
• IS: uses classifier confidence + marginal diversity; higher is better but ignores real data statistics and is easily gamed. FID preferred because it compares against real data distribution and penalizes mode collapse (2).

[
  {"claim":"KL for N(2,0.25)||N(0,1) = 2.318147","code":"mu=2.0; s2=0.25; val=(mu**2+s2-1-sympy.log(s2))/2; result=abs(float(val)-2.3181471805599454)<1e-6"},
  {"claim":"alpha_bar_3 = 0.36","code":"ab=0.9*0.8*0.5; result=abs(ab-0.36)<1e-12"},
  {"claim":"x_3 = 1.0 given x0=1, eps=0.5","code":"ab=0.36; x3=sympy.sqrt(ab)*1.0+sympy.sqrt(1-ab)*0.5; result=abs(float(x3)-1.0)<1e-9"},
  {"claim":"GAN optimal objective at p_g=p_data equals -2 ln 2","code":"v=sympy.log(sympy.Rational(1,2))+sympy.log(sympy.Rational(1,2)); result=abs(float(v)-(-1.3862943611198906))<1e-9"},
  {"claim":"CFG guided eps at w=3 is 1.10","code":"eu=0.20; ec=0.50; w=3; g=eu+w*(ec-eu); result=abs(g-1.10)<1e-9"}
]