World models and embodied AI: Question bank — AI-ML deep dive

Before we start, the symbols and names used everywhere below, in plain language:

Definition Anchors — states, actions, and the model

A world model is a learned "physics guesser": feed it the current situation (state $s_{t}$ ) and what you did (action $a_{t}$ ), and it predicts the next situation $s_{t + 1}$ and reward $r_{t}$ . Symbol shorthand: $\hat{P}_{θ} (s_{t + 1} ∣ s_{t}, a_{t})$ , where the hat means learned/approximate and $θ$ are the network weights.
Model-free means the agent never builds that guesser — it only learns what to do ( $π (a ∣ s)$ ) or how good a state is ( $V (s)$ ) from raw trial-and-error.
Rollout = starting from a state and repeatedly calling the model to imagine a chain of future states, like daydreaming several moves ahead.
Return $J$ = the total reward collected along a trajectory (a rollout), summed over the horizon: $J = \sum_{t} r (s_{t}, a_{t})$ . Because the model is random, we care about its expected value $E [J]$ — the average return over all imagined futures. $J_{sim}$ is that same return but computed from simulated (model-generated) rollouts rather than real ones.
MPC (Model Predictive Control) = a planning method: optimise a whole future action sequence in imagination, execute only the first action, then re-plan from the new state.

Definition Anchors — the Ha–Schmidhuber three-part architecture

The classic World Models agent has three named pieces (this is where the letters V, M, C come from):

V (Vision) = a VAE that compresses a raw observation into a small latent code $z_{t}$ .
M (Memory) = an RNN that predicts the next latent $z_{t + 1}$ ; because it is trained to output a probability distribution over $z_{t + 1}$ we write that distribution as $P_{M}$ (the "M" just labels which model — it is the same idea as $\hat{P}_{θ}$ , specialised to the latent space of M). $P_{M}$ and $\hat{P}_{θ}$ mean the same kind of object: a learned next-state predictor.
C (Controller) = a tiny policy that turns $z_{t}$ (and M's memory) into an action.
== $λ$ (lambda)== = a weight between 0 and 1 that says how much to trust simulated data. It appears in the Dyna policy update $θ_{π} \leftarrow θ_{π} + α [\nabla_{θ} J_{real} + λ \nabla_{θ} J_{sim}],$ where $α$ is the learning rate. Small $λ$ = lean on real experience; large $λ$ = trust the model's dreams.

True or false — justify

A world model must reconstruct the raw observation (e.g. every pixel) to be useful.

False — it only needs to predict task-relevant dynamics. Ha–Schmidhuber's model works in a compressed latent

z_{t}

from a VAE; pixel reconstruction is a training crutch for learning

z

, not the end goal.

Model-based RL is always more sample-efficient than model-free RL.

False — it is more efficient when the model is accurate. A badly-fit model injects biased synthetic data, and long rollouts compound its errors, so a good model-free method can beat a bad model-based one.

Learning

\hat{P}_{θ} (s_{t + 1} ∣ s_{t}, a_{t})

automatically gives you a good policy.

False — the model only tells you what happens, not what to do. You still need planning (MPC) or a policy trained inside the model; the model is a simulator, not a decider.

A single Gaussian output for the next latent state is fine for a racing car.

False — a car may turn left or right; a single Gaussian predicts their average (straight ahead), which is a state that never actually occurs. That is why M uses a mixture density network (several Gaussians).

Embodied AI is just RL with a robot attached.

False — the claim of embodiment is stronger: that structure of the body and its sensorimotor loop shapes what can be learned and how concepts get grounded. The morphology is part of the intelligence, not a peripheral.

In the Dyna framework, real and simulated experience are treated identically.

False — simulated experience is down-weighted by the trust weight

λ

(defined above) precisely because it comes from a fallible model; trusting it as much as real data would let model bias dominate.

A perfect world model removes all need for real-environment interaction.

Mostly false in practice — you still need some real data to fit and periodically correct the model, and to catch distribution shift when the policy visits states the model never trained on. In theory only, a perfect model plus reward function suffices.

The Markov property is an assumption we impose, not a fact about the world.

True — writing

s_{t + 1} \sim P (\cdot ∣ s_{t}, a_{t})

assumes the current state summarises all relevant history. When it does not (partial observability), we bolt on memory (the RNN hidden state

h_{t}

) to make the augmented state Markov-ish.

Spot the error

"We minimise

L_{M} = - lo g P_{M} (z_{t + 1})

, so we are teaching the model to make

z_{t + 1}

large."

Wrong — recall

P_{M}

is M's predicted distribution over the next latent. We maximise the probability M assigns to the observed

z_{t + 1}

, not the value of

z_{t + 1}

. The minus sign turns "maximise likelihood" into "minimise negative log-likelihood"; the target stays fixed.

"MPC picks the single best next action and ignores the future."

Wrong — MPC (Model Predictive Control, defined above) optimises a whole horizon

a_{0 : H - 1}

by simulating

H

steps ahead, then executes only the first action and re-plans. It very much accounts for the future; it just re-decides each step.

"Because the model is stochastic, we can just use one sampled trajectory to estimate the return

J

Wrong — one sample of a random quantity is a noisy estimate. We average over

N

sampled rollouts so that

J \approx \frac{1}{N} \sum_{k} \sum_{t} r (s_{t}^{(k)}, a_{t})

approaches the true expectation.

"Backpropagating through the model needs the model to be stochastic sampling."

Wrong — gradient-based planning needs the model to be differentiable w.r.t. states and actions. Raw sampling blocks gradients; you either use a differentiable/reparameterised model or fall back to gradient-free methods (CEM, CMA-ES).

"The controller C is small, so the whole architecture is weak."

Wrong — C (the tiny policy, defined above) is small because V and M did the hard work of compressing pixels into a clean latent and modelling dynamics. Operating on a good

z_{t}

makes a tiny linear policy sufficient; smallness is a feature, not a defect.

"Training C entirely inside M's dream is the same as imitation learning."

Wrong — imitation learning copies an expert's demonstrated actions. Dreaming trains C by reward maximisation inside M's learned simulator; there is no expert being copied.

"Since we learn dynamics, we never suffer from the sim-to-real gap."

Wrong — a learned model is a simulator, so it has its own reality gap: states the policy exploits may be model artefacts. This is the same failure mode studied in Sim-to-Real-Transfer, just with a neural sim instead of a physics engine.

Why questions

Why learn

\hat{P}

explicitly instead of just more

Q (s, a)

from experience?

Because an explicit model lets you generate new experience cheaply, plan by imagining futures, and reuse the same dynamics across different reward/goal tasks — a value function is tied to one task.

Why does the World Models paper use an RNN for M rather than a plain feed-forward net?

The next latent depends on history, not just the current frame (velocity, momentum, what is off-screen). The RNN's hidden state

h_{t}

carries that history so the prediction can be conditioned on context.

Why do Dyna rollouts stay short (

k

-step) instead of long?

Every step the model makes a small error; those errors compound multiplicatively over a rollout. Short rollouts keep the synthetic states close to reality so the generated data is still trustworthy.

Why is a mixture of Gaussians (not a diffusion or transformer) the classic choice for M?

For a low-dimensional latent, a mixture of

K

Gaussian components already captures multimodality cheaply and gives a closed-form log-likelihood loss. Richer stochastic models like Diffusion-Models or Vision-Transformers are used when the target is high-dimensional or long-range.

Why does embodiment help ground abstract concepts like "heavy"?

"Heavy" only means something through the effort of moving objects — the sensorimotor consequences. A disembodied text model has the word but not the felt dynamics, so its concept is ungrounded.

Why does the trust weight

λ

in the Dyna update matter more early in training?

Early on the model is poorly fit, so

\nabla_{θ} J_{sim}

is most biased; a small

λ

limits how much that bad gradient corrupts the policy until the model becomes reliable.

Why can partial observability actually encourage exploration in embodied agents?

With limited sensors, the agent must move to gather missing information — turning its head, walking around an object — and that information-seeking motion produces diverse experience for free.

Edge cases

What does the return integral

J

collapse to when the model

\hat{P}

is deterministic?

The expectation over next states vanishes: each

s_{t + 1}

is a fixed function of

(s_{t}, a_{t})

, so

J

becomes a plain sum along one trajectory with no averaging needed.

What happens to planning if the horizon

H = 1

MPC becomes greedy one-step reward maximisation — it ignores delayed consequences entirely, so it fails on any task where a good move now costs reward later (e.g. braking before a turn).

If the mixture uses

K = 1

component, what have we recovered?

A single-Gaussian forward model — unimodal, so it averages away branching futures. This is the degenerate case the mixture was introduced to fix.

The policy visits a state region the model never saw in training — what breaks?

The model extrapolates unreliably, often confidently predicting nonsense; the planner may then "hallucinate" high reward there and exploit the fantasy. This is model exploitation / out-of-distribution failure.

Zero reward everywhere over the horizon — is planning meaningless?

Yes for pure reward-driven MPC: every action sequence ties at

J = 0

, so the objective gives no signal. You would need intrinsic/curiosity rewards or a different objective to act sensibly.

What if the environment is truly non-Markov even after adding

h_{t}

Then

\hat{P}

conditions on an incomplete state and its predictions are systematically wrong; you need longer memory, attention over history, or explicit belief-state tracking to restore predictability.

As the VAE latent dimension

\to

the raw pixel dimension, what advantage is lost?

Compression — the whole point of V. With no dimensionality reduction, M and C must model raw high-dimensional data directly, losing the abstraction that made C tiny and training tractable.

Recall One-line summary to carry away

A world model is a learned simulator that buys sample-efficiency and planning only as far as it is accurate; embodiment claims that intelligence is shaped by the body–environment loop, not bolted on afterward. Every trap above is a variant of over-trusting a fallible model or under-appreciating that a simulator is not a decision-maker.