Exercises — World models and embodied AI

(a) is model-free. $Q(s,a)$ scores actions directly from trial-and-error; it never says what the world will do next. (b) is a world model. It predicts consequences — the "if I push this, it falls" knowledge. With it you can imagine futures before acting. Answer: (b).

False. Compression is done by V (Vision) — a Variational Autoencoder (Variational-Autoencoders) mapping observation $o_t$ to latent $z_t$. M predicts $z_{t+1}$ over time; V squeezes space, M predicts time.

• Forward model — predicts next state $s_{t+1}$ from $(s_t,a_t)$.
• MPC (Model Predictive Control) — searches action sequences to maximise imagined return, then executes the first action.
• Dyna — mixes real and model-simulated experience to update the policy.
• MDN (Mixture Density Network) — outputs a mixture of Gaussians so the model can predict a multimodal future (left OR right).

Expectation = probability-weighted average. Each has probability $\frac{1}{3}$: $\mathbb{E}[r]=\frac{1}{3}(2)+\frac{1}{3}(6)+\frac{1}{3}(10)=\frac{18}{3}=6.$ Answer: 6. This is exactly what integrating $\int \hat{P}(s_1\mid s_0,a_0)rd{s}_1$ collapses to when the distribution is a discrete set of outcomes.

$J={\sum }_{t=0}^{H-1}{r}_t=1+4+3=8.$ Answer: 8. MPC would compare this $J$ against other candidates and keep the largest, then execute only $a_0$ before re-planning.

$J=0.9^0(1)+0.9^1(4)+0.9^2(3)=1+3.6+0.9\cdot 0.9\cdot 3.$ $0.9^2=0.81$, so $0.81\times 3=2.43$. $J=1+3.6+2.43=\mathrm{7.03.}$ Answer: 7.03. Discounting shrinks later rewards, so the model trusts its own far-future guesses less — which is wise, since model errors compound.

$\hat{J}=\frac{1}{4}(5+9+1+5)=\frac{20}{4}=5.$ Answer: 5. We average because the true objective is the expectation over all possible futures (the model is stochastic). Taking the max would reward action sequences that got lucky on one rollout — over-optimistic bias. The mean is an unbiased estimate of ${\mathbb{E}}_{\hat{P}}[J]$.

$(1-\epsilon )=0.9$.

• $H=2:0.9^2=0.81$ → about $81\%$ reliable.
• $H=10:0.9^{10}\approx 0.3487$ → about $35\%$ reliable. Answer: $0.81$ vs $\approx 0.349$. Long imagined rollouts decay fast, so Dyna uses short $k$-step rollouts — where $k$ (the rollout length from the symbol dictionary) is kept small — starting from real states. You re-anchor to reality before the compounding error swamps the signal.

A single Gaussian is forced to place its mean at the average: $\mu =\frac{1}{2}(-2)+\frac{1}{2}(+2)=0.$ Answer: mean $=0$ — the middle of the road, which is a tree/wall. The model predicts an outcome that never actually happens. A mixture keeps two peaks (one at $-2$, one at $+2$), preserving the real bimodal choice. This is the whole reason M is a Mixture Density Network — see Recurrent-Neural-Networks for the RNN that feeds it context $h_t$.

Trace the chain of influence $a_0\to s_1\to s_2\to r_2$: $\frac{\partial J}{\partial a_0}=\frac{\partial r_2}{\partial s_2}\cdot \frac{\partial s_2}{\partial s_1}\cdot \frac{\partial s_1}{\partial a_0}.$ Numbers: $4\times 0.5\times 3=6.$ Answer: 6. This is exactly the "backprop through the model" gradient from the parent's formula callout — action $a_0$'s effect on reward flows through every future state by multiplying the per-step Jacobians. Related idea: gradient descent through a differentiable simulator underlies Sim-to-Real-Transfer.

Cost: $N_r\cdot 1+N_s\cdot 0.01=N_r+30N_r(0.01)=N_r(1+0.3)=1.3N_r\le 20$. So $N_r\le 20/1.3\approx 15.38$ → take $N_r=15$. Then $N_s=30\times 15=450$; total updates $=15+450=465$. That is below 500, so bump to $N_r=16$: cost $=1.3\times 16=20.8>20$ — over budget. So within budget the max is $N_r=15$, giving 465 updates. Answer: $N_r=15,N_s=450$, total $=465$ updates at cost $19.5$ units. The lesson: cheap simulated data (Reinforcement-Learning-Basics policy updates) multiplies your effective learning per expensive real sample — the core Dyna win.

Pipeline (V, M, C are defined in the symbol dictionary above):

1. V (Vision): Train a VAE (Variational-Autoencoders) on camera frames $o_t$ → latent $z_t\in {\mathbb{R}}^d$. Compresses pixels to "what matters." Could swap the encoder for a Vision-Transformers backbone for richer features.
2. M (Memory): An RNN + MDN predicting $P(z_{t+1}\mid z_t,a_t,h_t)={\sum }_i{\pi }_i\mathcal{N}({\mu }_i,{\sigma }_i^2)$. Loss = negative log-likelihood ${\mathcal{L}}_M=-\log ({\sum }_{i=1}^K{\pi }_i\mathcal{N}(z_{t+1};{\mu }_i,{\sigma }_i^2)).$ Mixture is needed for multimodal futures.
3. C (Controller): A tiny policy $a_t=W[z_t;h_t]+b$, trained inside M's dream — rollouts sampled from M, no robot needed.
4. Bootstrapping data: Seed V and M from the 5,000 demonstrations via Imitation-Learning, so you never start from random flailing on real hardware.
5. Safeguard against model exploitation: the controller can find fantasy action sequences that score high only because M is wrong there. Fixes: short $k$-step rollouts, a $\lambda$ weight downscaling simulated gradients (Dyna), and periodic re-collection of a few real trajectories to correct M where it drifts — the Sim-to-Real-Transfer loop. Answer: a V→M→C loop trained mostly in imagination, seeded by imitation, guarded by short rollouts + $\lambda$ trust + real recalibration.

$J_A=2+0.9(5)+0.81(5)=2+4.5+4.05=\mathrm{10.55.}$ $J_B=4+0.9(3)+0.81(2)=4+2.7+1.62=\mathrm{8.32.}$ $J_A>J_B$ by $10.55-8.32=2.23$. Answer: execute $a_0$ from sequence A; it wins by $2.23$. MPC then re-plans at the next real step (receding horizon), because the world is stochastic and the model is imperfect.