Worked examples — World models and embodied AI

This page takes the machinery from the parent note — the forward model ${\hat{P}}_{\theta }(s_{t+1}\mid s_t,a_t)$, planning with it, and the V-M-C dream architecture — and drives it through every kind of input you can hit. Before we start, one promise from the contract: no symbol appears before we say what it means in words and show it in a picture.

Words we will use, each earned here:

• state $s_t$ — a full snapshot of the situation at time step $t$ (the little subscript $t$ just means "at time step number $t$"; $t=0$ is now, $t=1$ is one tick later).
• action $a_t$ — the choice the agent makes at step $t$.
• reward $r_t$ — a single number saying "how good was that." Bigger is better.
• latent $z_t$ — a compressed state: instead of a whole 64×64 picture, a short list of numbers that keeps only what matters.
• rollout — running the dream machine forward several steps, feeding its own output back into itself.

---

The scenario matrix

Every world-model problem is one of these cells. The 8 examples below hit all of them.

Cell	What is stressed	Degenerate / limiting question it answers
C1 Deterministic dynamics	model is a plain function, no randomness	what happens when $\sigma \to 0$?
C2 Stochastic single mode	one bell-curve of futures	how does uncertainty grow per step?
C3 Multimodal future	future forks (left OR right)	why one Gaussian fails
C4 Zero / no-op action	action does nothing	does the model leave the state alone?
C5 Planning horizon $H$	choosing an action sequence by return	short vs long horizon trade-off
C6 Compounding model error	dream drifts from reality	why $k$-step rollouts, not $\infty$-step
C7 Dyna real+sim mixing	weighting simulated data	the $\lambda$ trust knob
C8 Real-world word problem	robot arm, sim-to-real	grounding + Sim-to-Real-Transfer

---

The building blocks, in plain words

---

Worked examples

Example 1 — Cell C1: deterministic dynamics ($\sigma \to 0$)

Forecast: guess $z_3$ and whether $J$ is near zero or very negative.

1. Roll the model forward. Why this step? With $\sigma =0$ the bell curve collapses to a spike at $\mu$, so the "distribution" is really a plain function — no averaging needed. $z_1=2+1=3,z_2=3+1=4,z_3=4-3=1.$
2. Collect rewards at $t=0,1,2$. Why this step? $J={\sum }_{t=0}^{H-1}{r}_t$ uses steps $\mathrm{0..}H-1$, i.e. the states before each action lands. $r_0=-|2|=-2,r_1=-|3|=-3,r_2=-|4|=-4.$
3. Sum. $J=-2-3-4=-9.$

Verify: every $z$ is an exact integer because there is no noise — that is the signature of a deterministic model ($\sigma \to 0$). $z_3=1$, $J=-9$. Units: reward is unitless "points", summed → still points. ✔

---

Example 2 — Cell C2: single-mode stochastic, uncertainty growth

Forecast: does uncertainty stay at $1$, grow to $2$, or grow to $4$?

1. Write each step as start + noise. Why? Independent noises add, and for independent randoms the variances add (not the $\sigma$'s directly). $z_1=0+{\epsilon }_1,z_2=z_1+{\epsilon }_2={\epsilon }_1+{\epsilon }_2.$
2. Add variances. Why this rule and not adding $\sigma$? Because $\sigma$ is a length and variances are the thing that superimposes for independent noise — like combining independent wobbles by Pythagoras, not by simple addition. $\mathrm{Var}(z_2)={\sigma }^2+{\sigma }^2=1+1=2.$
3. So the spread is $\sqrt{2}\approx 1.414$.

Verify: at step $1$ variance was $1$; at step $2$ it is $2$; it grows linearly with steps. This is exactly why long dream-rollouts get untrustworthy (foreshadows C6). ✔

---

Example 3 — Cell C3: multimodal future (why one Gaussian fails)

Forecast: will the single Gaussian sensibly predict "left or right", or predict something nonsensical?

1. A single Gaussian's mean is the average of the data. Why? Maximum likelihood for one Gaussian sets $\mu$ to the sample mean — that is what minimising $\sum (z-\mu {)}^2$ does. $\mu =\frac{1}{2}(-4)+\frac{1}{2}(+4)=0.$
2. Interpret $\mu =0$. The car predicts it will drive straight into the wall between the roads — a state that never actually happens.
3. The fix: a mixture with two components ${\mu }_1=-4,{\mu }_2=+4,{\pi }_1={\pi }_2=0.5$. Now the model puts probability mass on the real outcomes, none in the middle. This is precisely why the M-network in the parent uses a Mixture Density Network over a plain RNN regressor.

Verify: the mixture assigns $P(z\approx 0)\approx 0$ while the single Gaussian peaks there — the single model is confidently wrong. ${\mu }_{\text{single}}=0$. ✔

---

Example 4 — Cell C4: zero / no-op action (degenerate input)

Forecast: guess whether a tiny $0.02$ is harmless.

1. What SHOULD happen? Why check this? A correct model must satisfy: do nothing → state unchanged, i.e. ${\hat{z}}_{t+1}=z_t$. This is the degenerate anchor every dynamics model must pass.
2. What DOES happen? Each step adds $0.02$. Over $50$ steps the drift is $50\times 0.02=1.0$.
3. Judge. A one-unit phantom drift with zero action means the model hallucinates motion. In a dream-trained controller this is catastrophic: the policy "learns" to counter a force that does not exist.

Verify: ideal drift $=0$; observed $=1.0\ne 0$ → model fails the no-op test. The number $1.0$ is the accumulated bias $50\times 0.02$. ✔

---

Example 5 — Cell C5: planning over a horizon $H$ (MPC)

Forecast: can the agent reach and stay rewarded within 3 steps?

1. Enumerate — it's cheap for small $H$. Why brute force? With $2^3=8$ sequences we can afford exact search; this is "shooting" MPC in its simplest form.
2. Score two candidates.

• $(+1,+1,-1)$: states visited at $t=0,1,2$ are $0,1,2$ → rewards $0,0,+1$ → $J=1$.
• $(+1,+1,+1)$: states $0,1,2$ → rewards $0,0,+1$ → also $J=1$ (the reward is checked at $s_t$ before $a_t$ moves it, so step-2's $s_2=2$ scores).

3. Best return. No sequence beats $J=1$ because you cannot reach $s=2$ before $t=2$ from $s_0=0$ (minimum two $+1$ steps), so at most one rewarded step fits in the window.

Verify: minimum steps to reach $s=2$ is $2$; window has steps $\{0,1,2\}$, so exactly one visit to $s=2$ is possible → $J_{\max }=1$. Lengthening $H$ would allow lingering and higher return — the horizon caps achievable return. ✔

---

Example 6 — Cell C6: compounding error → short rollouts

Forecast: guess the $10$-step predicted value — is $10\%$ per step "small"?

1. Model the compounding. Why multiply not add? Errors in an autoregressive model feed back into the next input, so a fractional bias compounds geometrically: predicted $=1.0\times 1.1^k$.
2. Evaluate. $k=1:1.1^1=1.1$ (error $10\%$). $k=10:1.1^{10}\approx 2.594$ (error $\approx 159\%$).
3. Decision. Short $k$-step rollouts keep dream error tiny; long rollouts explode. This is the exact reason Dyna uses short simulated rollouts, established in the parent's Step 3.

Verify: $1.1^{10}=2.5937\dots$ so a modest $10\%$ step bias becomes a $>2.5\times$ blowup in ten steps — dream is untrustworthy long-range. ✔

---

Example 7 — Cell C7: Dyna real + sim weighting ($\lambda$)

Forecast: guess the update, and whether cutting $\lambda$ to $0$ throws away useful data.

1. Plug in. Why the $\lambda$ multiplier? It scales how much we believe the dream; a poor model → shrink $\lambda$ so its bias barely moves us. $\Delta \theta =0.1(4+0.5\times 10)=0.1(4+5)=0.1\times 9=\mathrm{0.9.}$
2. Limit $\lambda \to 0$. $\Delta \theta \to 0.1\times 4=0.4$ — we fall back to pure model-free RL: only real experience counts. Safe but sample-hungry.
3. Limit $\lambda \to \infty$. The (possibly wrong) dream dominates — fast but can diverge.

Verify: with $\lambda =0.5$, $\Delta \theta =0.9$; with $\lambda =0$, $\Delta \theta =0.4$. The knob smoothly trades speed for safety. ✔

---

Example 8 — Cell C8: real-world word problem (robot + sim-to-real)

Forecast: will the transfer succeed or slip?

1. Dream target. Controller aims for reading $=5$ (dream's grip threshold). Why $=5$? It was trained to hit the minimum successful force in simulation.
2. Apply the reality gap. Why multiply by $0.8$? The real sensor under-reports; the effective real force is $0.8\times 5=4.0$.
3. Compare to the true threshold $5$. $4.0<5$ → the grip fails (slips). This is a textbook Sim-to-Real-Transfer failure: the dream was internally consistent but ungrounded.
4. Fix. Domain randomisation / calibration so the model's latent matches real physics — the grounding argument from the parent's Embodied-AI section.

Verify: real force $=0.8\times 5=4.0<5$ → fails. If instead the controller over-commands to reading $6.25$, real force $=0.8\times 6.25=5.0\ge 5$ → succeeds. The gap is exactly the $0.8$ factor. ✔

---

Recall Self-test

Model with $\sigma \to 0$ behaves like what? ::: A plain deterministic function (bell collapses to a spike). Two independent noise steps of $\sigma =1$ give what variance after two steps? ::: $2$ (variances add). Why a mixture, not one Gaussian, for a car at a fork? ::: One Gaussian predicts the average (straight into the wall); a mixture keeps mass on the real left/right outcomes. What must a correct model output for a no-op action? ::: The unchanged state, ${\hat{z}}_{t+1}=z_t$. Why does Dyna prefer short $k$-step rollouts? ::: Model error compounds geometrically over steps. What does $\lambda \to 0$ reduce Dyna to? ::: Pure model-free RL (only real experience). A $0.8$ sensor gap turns a dream force of $5$ into what real force? ::: $4.0$ — below threshold, grasp fails.