Model-based RL overview: Question bank — AI-ML deep dive

Before we start, one shared vocabulary check so no symbol or acronym is unearned:

Recall Symbols and acronyms used on this page

MBRL = Model-Based Reinforcement Learning: the agent first learns (or is given) a model of the world, then plans inside that model instead of only trying actions in reality.
MPC = Model Predictive Control: a planning style where, at each step, you search for the best short action sequence inside the model, execute only the first action, observe reality, then re-plan. See Model Predictive Control (MPC).
POMDP = Partially Observable Markov Decision Process: the agent sees only a partial observation of the true state, not the state itself.
$s$ = state (the situation the agent is in). $a$ = action (what it does). $s^{'}$ = the next state.
$\hat{f} (s, a)$ = the learned dynamics model: a function guessing the next state from the current one and the action. The hat means "estimated, not the true world."
$\overset{r}{^} (s, a)$ = the learned reward model: a guess of the reward you'd get.
$\hat{P} (s^{'} ∣ s, a)$ = probability the model assigns to landing in $s^{'}$ .
$H$ = horizon, the number of steps you imagine ahead before stopping.
$Δ s = s^{'} - s$ = the change in state, which we often predict instead of $s^{'}$ itself.
$γ$ = discount factor between 0 and 1; future reward matters less the further away it is.

Two pictures anchor the traps below. The first shows why a tiny per-step model error becomes a huge plan error (the compounding enemy). The second shows why we regress the small change $Δ s$ instead of the large raw next state $s^{'}$ .

Keep both in mind: nearly every "concept trap" on this page is really one of these two pictures in disguise.

True or false — justify

MBRL is always more sample-efficient than model-free RL

Usually, not always. It is efficient when the dynamics are learnable; on hard-to-model, chaotic dynamics a bad model can waste real data by making the agent plan against a fantasy — the red curve in the first figure blows up. See Sample Efficiency and Model-free RL.

A perfect model of the environment guarantees optimal behaviour

No. A perfect model is only a perfect simulator — it tells you what would happen, but not what you should do. Extracting the best action still requires searching an astronomically large tree of possible futures (in chess the rules are exact, yet the number of games is larger than atoms in the universe), so the planning/search problem stays hard even with a flawless model. See Bellman Equation.

If the model is given (rules known), there is nothing left to learn

False. When the model is given, the learning of dynamics disappears but the difficulty simply moves to planning/search: deciding which futures to expand and how deep to look. That search — choosing where to spend limited compute in a huge tree — is the entire game in systems like AlphaZero.

A learned dynamics model must reproduce every pixel of the observation

False. It only needs to be accurate about what affects decisions and reward. Fitting every irrelevant pixel wastes capacity and can even hurt; value-aware / latent models (MuZero and Dreamer) deliberately ignore reward-irrelevant detail and predict only what changes the decision.

Planning inside the model costs real environment interactions

False — that is the entire point. Imagined rollouts

(s, a) \to (\overset{r}{^}, \overset{s}{^}^{'})

are cheap compute; real steps (a robot arm that can break, a live trade that can lose money) are the expensive resource being conserved.

Longer planning horizons always give better plans in MBRL

False. Longer horizons let per-step model error compound (the widening red gap in the first figure), so beyond some point the imagined future is pure fiction; that is why MPC re-plans every step with a short horizon. See Model Predictive Control (MPC).

The least-squares model loss is chosen arbitrarily for convenience

False — it is derived. Assuming Gaussian next-state noise, maximum likelihood reduces exactly to minimizing

∥ s^{'} - \hat{f} (s, a) ∥^{2}

; the squared error is a consequence of that noise assumption, not a design choice.

In Dyna, imagined transitions can substitute for real ones during value updates

True — imagined

(s, a) \to (\overset{r}{^}, \overset{s}{^}^{'})

tuples feed extra Q-learning updates, amplifying scarce real data. But they inherit the model's errors, so a bad model corrupts the value estimate. See Dyna Architecture.

Predicting

Δ s

vs predicting

s^{'}

changes what the model can represent

False — they are algebraically equivalent (add

s

back). It changes how easily it learns: as the second figure shows, the residual

Δ s

is small while raw

s^{'}

sits on a big near-identity line, so regressing

Δ s

is more stable and accurate.

Spot the error

"We fit

\hat{f}

once at the start, then plan forever inside it."

The error: dynamics coverage. Early data only covers early states; as the policy improves it visits new states the model never saw. MBRL must keep collecting real data and re-fitting (the loop Act→Learn→Plan→repeat).

"Our model has low training loss, so our planned actions will be good."

Low one-step prediction error does not bound multi-step rollout error — errors accumulate over

H

steps roughly like

H ϵ

or worse (first figure). Good one-step fit ≠ good long-horizon plan.

"MPC computes the best action sequence

a_{0 : H}

and then executes all of it."

MPC executes only

a_{0}

, then observes the real next state and re-plans. Executing the whole sequence would blindly trust an imagined future that has already drifted from reality.

"We used a single neural net for the model, so we know how confident it is."

A single model gives a point prediction with no honest uncertainty. To know where the model is untrustworthy you need ensembles or uncertainty estimates, which flag risky, poorly-modelled regions.

"The Bellman backup inside the model needs real rewards to work."

No — the whole point is that

\overset{r}{^}

and

\hat{P}

come from the model, so

V^{*} (s) = max_{a} [\overset{r}{^} (s, a) + γ \sum_{s^{'}} \hat{P} (s^{'} ∣ s, a) V^{*} (s^{'})]

is evaluated with zero real interaction.

"Model-based means the agent never uses a value function."

Wrong — many MBRL methods (Dyna, Dreamer) still learn a value/policy; the model feeds those model-free updates with imagined data instead of replacing them.

Why questions

Why does maximum likelihood under Gaussian noise become squared-error regression?

Because

lo g p (s^{'} ∣ s, a) = - \frac{1}{2 σ ^{2}} ∥ s^{'} - \hat{f} (s, a) ∥^{2} + const

; maximizing the log-likelihood over data is identical to minimizing the sum of squared residuals.

Why predict the change

Δ s

instead of the raw next state

s^{'}

States usually change slowly, so

s^{'} \approx s

and the raw target has a huge near-identity component the network wastes capacity fitting; the residual

Δ s

is small and easier, giving stable, accurate learning (second figure).

Why does MBRL re-plan at every timestep instead of trusting one plan?

Because model error compounds along a rollout; re-planning from the real observed state resets the error and stops the imagined trajectory from drifting into fiction.

Why do we build a model at all instead of just acting more (model-free)?

Real interaction is expensive and sometimes dangerous; a model lets you imagine consequences cheaply, trading extra computation for far fewer real-world trials — the sample-efficiency payoff.

Why can a slightly wrong model produce a wildly wrong plan?

A planner exploits the model — it seeks actions the model rates highly, which can be exactly where the model is over-optimistically wrong; small errors get selected and amplified over the horizon.

Why do latent/value-aware models (Dreamer, MuZero) often beat pixel-perfect models?

They spend capacity only on features that predict reward and value, ignoring reward-irrelevant detail, so they are more accurate where accuracy actually changes the decision. See MuZero and Dreamer.

Why is the discount

γ

inside the planning objective, not just in real learning?

Planning evaluates the same Bellman value as the real problem, so it must down-weight distant imagined rewards identically — otherwise the plan optimizes a different objective than the true task.

Edge cases

If the environment dynamics are deterministic, do we still need

\hat{P} (s^{'} ∣ s, a)

No — a deterministic

s^{'} = \hat{f} (s, a)

suffices; the distribution

\hat{P}

collapses to a single outcome. Probabilistic models matter when the world is stochastic.

What happens to MBRL when the environment is not Markov (history matters)?

The core assumption of the Markov Decision Process breaks, so a model conditioned only on

(s, a)

mispredicts; you must augment the state (e.g. stack history or use a recurrent latent) to restore the Markov property.

How is a POMDP a distinct failure mode from generic non-Markov dynamics?

In a POMDP the true state is Markov, but you never observe it directly — you see only a partial observation. So the dynamics aren't broken; your inputs are. The fix is different: infer a belief/latent state (a summary of past observations) and model dynamics in that space, rather than just stacking a few past frames. See MuZero and Dreamer.

Concretely, why can a single observation mislead the model in a POMDP?

Two genuinely different underlying states can produce the same observation (e.g. a velocity you can't see from one frame), so the model would predict the same next state for both and be wrong for at least one — only accumulated history disambiguates them.

With a perfect model and zero noise, how many real steps does planning need to find the optimum?

Zero extra real steps for planning itself — all search happens inside the model. You still needed real data to fit the model (or it was given), but no real interaction is spent evaluating candidate futures.

What is the failure mode when the horizon

H

is set to 1?

The agent becomes greedy and myopic — it optimizes only the immediate imagined reward and can walk into states that are locally good but globally bad, losing the main advantage of planning ahead.

What is the failure mode when

H

is set very large?

Compounding model error dominates (first figure); the far end of the rollout is essentially noise, so the plan optimizes a hallucinated future — the reason short horizons plus re-planning (MPC) are preferred.

If real data is plentiful and cheap, does MBRL still help?

Its main edge (sample efficiency) shrinks, so plain Model-free RL may be simpler and safer — you avoid model-error bias entirely. MBRL earns its keep when real interaction is the scarce resource.

What happens in Dyna if the model is systematically biased?

Every imagined update injects the same bias, so more imagined data makes the value estimate confidently wrong faster — a case where extra "data" hurts rather than helps.

A region of state space was never visited during data collection — what does the model do there?

It extrapolates with no grounding, often over-confidently; planners can be lured into these blind spots, which is why uncertainty-aware models penalize acting where the model is untrained. See Uncertainty and Ensembles in RL.