Exercises — Reparameterization trick

(A) blocks it. The arrow "$\sim$" (sampling) is a black box: if you nudge ${\mu }_{\varphi }$, the drawn number jumps around randomly, so $\frac{\partial z}{\partial {\mu }_{\varphi }}$ has no defined value. In (B) the sampling happens first into $ϵ$ (which does not touch $\varphi$), and then $z$ is an ordinary formula of $\varphi$. So $\frac{\partial z}{\partial {\mu }_{\varphi }}=1$ and $\frac{\partial z}{\partial {\sigma }_{\varphi }}=ϵ$ are perfectly ordinary derivatives. (B) is differentiable.

$z=\mu +\sigma ϵ=4+2\times (-1.5)=4-3=1.$ Picture: start at the centre $4$, step $\sigma ϵ=-3$ (left, because $ϵ<0$), land at $1$.

$z\sim \mathcal{N}(\mu ,{\sigma }^2)$ by the location–scale property.

• Mean: $\mathbb{E}[z]=\mu +\sigma \cdot 0=\mu$.
• Variance: $\mathrm{Var}(z)={\sigma }^2\mathrm{Var}(ϵ)={\sigma }^2\cdot 1={\sigma }^2$. "Location" = the shift $\mu$; "scale" = the stretch $\sigma$.

Step 1 — build $z$. $z=\mu +\sigma ϵ=1.0+0.8\times 0.5=1.4$. Step 2 — loss. $f=(1.4-2{)}^2=(-0.6{)}^2=0.36$. Step 3 — $\frac{\partial f}{\partial z}$. $f=(z-2{)}^2\Rightarrow \frac{\partial f}{\partial z}=2(z-2)=2(-0.6)=-1.2$. Step 4 — local derivatives of the pipe. $\frac{\partial z}{\partial \mu }=1,\frac{\partial z}{\partial \sigma }=ϵ=0.5$. Step 5 — chain rule. $\frac{\partial f}{\partial \mu }=-1.2\times 1=-1.2,\frac{\partial f}{\partial \sigma }=-1.2\times 0.5=-\mathrm{0.6.}$ Read it: both gradients are negative, so gradient descent will increase $\mu$ and $\sigma$ — moving $z$ up toward the target $2$. Makes sense: $z=1.4$ is below target.

Element-wise: $z_1=0+1\times 2=2,z_2=3+2\times (-1)=1.$ So $z=(2,1)$. The $\odot$ just says "multiply entry-by-entry" — dimension 1 uses its own ${\sigma }_1,{ϵ}_1$, dimension 2 its own. No cross-talk.

Predict: $\frac{\partial f}{\partial \sigma }=\frac{\partial f}{\partial z}\cdot ϵ$. Here $ϵ<0$. First compute $z=1.0+0.8\times (-1.0)=0.2$, so $\frac{\partial f}{\partial z}=2(0.2-2)=-3.6<0$. Negative times negative $\Rightarrow$ positive. Compute: $\frac{\partial f}{\partial \sigma }=-3.6\times (-1.0)=3.6$. Why the flip: with $ϵ<0$, growing $\sigma$ pushes $z$ down, away from target — so loss rises, hence a positive gradient telling us to shrink $\sigma$.

Per-sample gradient: $z^{(s)}=0.5+{ϵ}^{(s)}$, and $\frac{\partial f}{\partial \mu }=2(z^{(s)}-2)\cdot 1=2(0.5+{ϵ}^{(s)}-2)=2({ϵ}^{(s)}-1.5)$.

• $ϵ=1$: $2(1-1.5)=-1.0$
• $ϵ=-0.5$: $2(-0.5-1.5)=-4.0$
• $ϵ=0.2$: $2(0.2-1.5)=-2.6$

Average: $\hat{g}=\frac{-1.0-4.0-2.6}{3}=\frac{-7.6}{3}=-2.5\overline{3}\approx -2.533$. Exact: $\mathbb{E}[2(ϵ-1.5)]=2(\mathbb{E}[ϵ]-1.5)=2(0-1.5)=-3.0$. Read it: the 3-sample estimate $-2.533$ hovers near the true $-3.0$; with more samples it converges (law of large numbers). Both point the same direction: increase $\mu$.

Write the reparameterized loss as an integral: $\int p(ϵ)f(g_{\varphi }(ϵ))dϵ$. The measure $p(ϵ)dϵ$ does not contain $\varphi$, so by the Leibniz rule you may pull ${\nabla }_{\varphi }$ inside — only $f(g_{\varphi }(\cdot ))$ depends on $\varphi$. For $\int q_{\varphi }(z)f(z)dz$ the density itself $q_{\varphi }$ carries $\varphi$. Differentiating produces an extra term $\int ({\nabla }_{\varphi }{q}_{\varphi })fdz$ — the score-function term you meet in 6.1.02-Policy-Gradients. Reparameterization avoids that by moving all $\varphi$-dependence into the integrand and none into the measure.

Transform: $u=a+(b-a)\eta$, where $\eta \sim \mathcal{U}(0,1)$ is parameter-free. Check endpoints: $\eta =0\Rightarrow u=a$; $\eta =1\Rightarrow u=b$. Uniform stays uniform under a linear map, so $u\sim \mathcal{U}(a,b)$. ✓ Derivatives: $\frac{\partial u}{\partial a}=1-\eta ,\frac{\partial u}{\partial b}=\eta$. Same recipe as the Gaussian: fixed base noise, linear deterministic pipe.

The reconstruction term is an expectation over $z$, so it needs a differentiable sample — that's where the trick lives. The 4.3.01-KL-Divergence term between two Gaussians has a closed form: $\text{KL}(\mathcal{N}(\mu ,{\sigma }^2)\Vert \mathcal{N}(0,1))=\frac{1}{2}({\mu }^2+{\sigma }^2-\ln {\sigma }^2-1),$ which is an ordinary function of $\mu ,\sigma$ — no sampling, already differentiable. See the full breakdown in 4.5.02-ELBO-Derivation. So the trick is only needed on the sampled expectation.

Formula: $\frac{1}{2}({\mu }^2+{\sigma }^2-\ln {\sigma }^2-1)$.

• $\mu =0,\sigma =1$: $\frac{1}{2}(0+1-\ln 1-1)=\frac{1}{2}(0)=0$. ✓ (identical distributions ⇒ zero divergence.)
• $\mu =1,\sigma =1$: $\frac{1}{2}(1+1-0-1)=\frac{1}{2}(1)=0.5$. Sanity: a shift with equal spread costs a small, positive KL.

The Gaussian trick works because a continuous invertible map $z=\mu +\sigma ϵ$ moves probability mass smoothly. For a discrete $z$, any deterministic function of $ϵ$ that outputs $\{0,1\}$ is a step function: its derivative w.r.t. $p_{\varphi }$ is $0$ almost everywhere and undefined at the jump. No smooth pipe exists. Escape route: relax the discreteness with the 5.2.03-Gumbel-Softmax trick — reparameterize with Gumbel noise plus a temperature $\tau$ that softens the step into a differentiable curve, then anneal $\tau \to 0$. Alternatively use the score-function / 6.1.02-Policy-Gradients estimator (higher variance, but works for any distribution).

Forward. $\sigma =e^0=1$. $z=\theta +\sigma ϵ=1+1\times 1=2$. $f=z^2=4$. Backward. $\frac{\partial f}{\partial z}=2z=4$.

• $\theta$: $\frac{\partial z}{\partial \theta }=1\Rightarrow \frac{\partial f}{\partial \theta }=4\times 1=4$.
• $\rho$: chain through $\sigma$. $\frac{\partial z}{\partial \sigma }=ϵ=1$, and $\frac{\partial \sigma }{\partial \rho }=e^{\rho }=1$. So $\frac{\partial f}{\partial \rho }=\frac{\partial f}{\partial z}\cdot \frac{\partial z}{\partial \sigma }\cdot \frac{\partial \sigma }{\partial \rho }=4\times 1\times 1=4$. Note the $e^{\rho }$ parametrization: it guarantees $\sigma >0$ and inserts an extra factor $\frac{\partial \sigma }{\partial \rho }=\sigma$ in the chain. This is standard 2.4.01-Backpropagation bookkeeping.

(R) usually wins. Reparameterization routes the gradient through $f$'s own slope $\frac{\partial f}{\partial z}$ — it uses information about how $f$ changes, so smooth $f$ gives smooth, low-variance gradients. The score-function estimator only sees $f$'s value multiplied by a log-density gradient; it ignores $f$'s slope and its variance grows with $f$'s magnitude and the tails of $\log q_{\varphi }$. Hence practitioners prefer (R) whenever the latent is continuous and $f$ is differentiable, falling back to (S) only for discrete/black-box $f$.

The reparameterization trick moves randomness into a ==parameter-free noise variable $ϵ$ so the sample becomes a deterministic function of the parameters==. The pipe for a Gaussian is ::: $z=\mu +\sigma ϵ$ with $ϵ\sim \mathcal{N}(0,1)$. The gradient we can now compute is ::: $\frac{\partial f}{\partial \mu }=\frac{\partial f}{\partial z}$ and $\frac{\partial f}{\partial \sigma }=\frac{\partial f}{\partial z}ϵ$. For discrete latents we switch to ::: Gumbel-Softmax or a score-function estimator.