Visual walkthrough — Model-based RL overview

We build one idea per step. Look at the coloured arrows in each figure — the text points at them.

---

Step 1 — What is a "model" doing, as a picture?

WHAT. A model is a machine that takes where you are now and what you do, and guesses where you end up next. We write your current situation as $s$ (a state — think "the full snapshot of the world right now": the ball's position, the robot's joint angles). We write your choice as $a$ (an action — a button you press, a push you apply). The true world has a rule $f$ that turns $(s,a)$ into the real next state, written $s^{\prime }$ (read "s-prime", meaning "the next s"). Our learned copy of that rule is written $\hat{f}$ (read "f-hat"; the little hat $\hat{}$ always means "our estimate of", not the real thing).

WHY. Before we can talk about errors, we must see clearly that there are two arrows leaving every point: the true one and our guessed one. The gap between their tips is the whole story.

PICTURE. In the figure, a single state is a dot. The blue arrow is the truth $s^{\prime }=f(s,a)$. The pink arrow is our model ${\hat{s}}^{\prime }=\hat{f}(s,a)$. The short yellow segment between the two arrowheads is the one-step error — we will name it in the next step.

---

Step 2 — Name the one-step error $ϵ$

WHAT. The yellow gap from Step 1 deserves a name. Call its length $ϵ$ (the Greek letter "epsilon", by long tradition the symbol for "a small error"). In general the gap is different at different places and choices, so it really depends on $(s,a)$; to get a single number that covers the whole rollout we take the worst case over all states and actions (the symbol $\sup$ below means "the largest value of", read "supremum"):

$\underset{\text{worst-case one-step error}}{\underbrace{ϵ}}={\sup }_{s,a}\Vert \underset{\text{our guess}}{\underbrace{\hat{f}(s,a)}}-\underset{\text{the truth}}{\underbrace{f(s,a)}}\Vert .$

Term by term: the two vertical-bar pairs $\Vert \cdots \Vert$ mean "the length of" (the norm — how far apart two points are, measured with an ordinary ruler). Inside, we subtract the true next state from the predicted one. The ${\sup }_{s,a}$ out front says "take the biggest such gap anywhere". The result $ϵ$ is a single non-negative number: the largest a single prediction can be off by.

WHY. We need one number we can carry through the whole derivation, not a value that silently changes at every state. Using the worst case makes every later inequality a safe upper bound. Regression training (the parent's least-squares fit) makes this $ϵ$ small — but it is never exactly zero on new inputs.

PICTURE. The figure zooms into one dot and draws the two arrowheads close together, with the yellow ruler $ϵ$ measuring the gap. A tiny $ϵ$ looks harmless here — that is the trap the next steps spring.

---

Step 3 — Imagining means feeding the guess back in (with the SAME actions)

WHAT. To plan, we don't stop at one step. We take our predicted next state ${\hat{s}}_1$ and feed it back into the same model to get ${\hat{s}}_2$, then ${\hat{s}}_3$, and so on. This is a rollout: a chain of imagined states. We index time with a subscript $t$, so ${\hat{s}}_t$ is "the imagined state $t$ steps into the future", and ${\hat{s}}_0=s_0$ is the real starting point (both agree, because we start from a real observation).

There are two chains, and they use the same fixed action sequence $a_0,a_1,a_2,\dots$ — the actions we plan to actually take. The true world evolves by its true rule $f$, and our imagination evolves by our learned rule $\hat{f}$:

$\underset{\text{true trajectory}}{\underbrace{s_{t+1}=f(s_t,a_t)}},\underset{\text{imagined trajectory}}{\underbrace{{\hat{s}}_{t+1}=\hat{f}({\hat{s}}_t,a_t)}}.$

Notice the same $a_t$ appears in both lines. This matters: we are comparing "what really happens if I take this plan" against "what my model thinks happens if I take this plan". If the actions differed, the two trajectories would drift apart for a boring reason (different choices) instead of the interesting reason (model error). Keeping $a_t$ identical isolates the model's mistake.

The dangerous word is back in: at step $t=1$ we no longer feed the true $s_1$ into the model — we feed our already-wrong ${\hat{s}}_1$. So the model is now guessing about a place the real world never even visited.

WHY. This feedback is exactly what makes imagination cheap (no real environment steps). But it is also why errors don't just sit there — they feed each other.

PICTURE. Two parallel chains march to the right under one shared action tape $a_0,a_1,\dots$: the blue true trajectory $s_0\to s_1\to s_2\to \dots$ and the pink imagined trajectory ${\hat{s}}_0\to {\hat{s}}_1\to {\hat{s}}_2\to \dots$. They start on the same dot; the yellow gaps between corresponding dots get visibly wider each step.

---

Step 4 — Why the error grows: the two sources add

WHAT. Let $E_t=\Vert {\hat{s}}_t-s_t\Vert$ be the total gap after $t$ imagined steps (capital $E$ for the accumulated error, to distinguish it from the single-step $ϵ$). Here $s_t$ is the true state from the true recursion $s_{t+1}=f(s_t,a_t)$ of Step 3, and ${\hat{s}}_t$ is the imagined state from ${\hat{s}}_{t+1}=\hat{f}({\hat{s}}_t,a_t)$ — same actions, so $E_t$ measures only the model's fault.

Now watch one step. Compare ${\hat{s}}_{t+1}=\hat{f}({\hat{s}}_t,a_t)$ against $s_{t+1}=f(s_t,a_t)$. Insert the true rule applied to the wrong input, $f({\hat{s}}_t,a_t)$, as a stepping stone (we add and subtract the same quantity, which changes nothing). Now the gap is split by the triangle inequality — the plain fact that "the straight distance from A to C is never more than going via a detour point B", written $\Vert A-C\Vert \le \Vert A-B\Vert +\Vert B-C\Vert$. With $A=\hat{f}({\hat{s}}_t,a_t)$, $B=f({\hat{s}}_t,a_t)$, $C=f(s_t,a_t)$:

$E_{t+1}=\Vert \hat{f}({\hat{s}}_t,a_t)-f(s_t,a_t)\Vert \le \underset{\text{fresh model error}\le ϵ}{\underbrace{\Vert \hat{f}({\hat{s}}_t,a_t)-f({\hat{s}}_t,a_t)\Vert }}+\underset{\text{how }f\text{ spreads the old gap}}{\underbrace{\Vert f({\hat{s}}_t,a_t)-f(s_t,a_t)\Vert }}.$

The first piece is a one-step model error (same ${\hat{s}}_t$ into both rules), so it is at most $ϵ$ (Step 2). The second piece asks: if I feed the true dynamics $f$ two inputs that differ by $E_t$, how far apart are the outputs? That is exactly a question about whether $f$ stretches or shrinks distances.

WHY. The triangle inequality is what lets us separate the model's fresh mistake from the inherited one — without it we could not bound a two-cause error by two nameable pieces. Now we must name how much $f$ can stretch a gap, because that factor decides everything. Call it $L$ (the Lipschitz constant — the largest amount $f$ can multiply any input distance by):

$\Vert f(x,a)-f(y,a)\Vert \le L\Vert x-y\Vert \text{for all }x,y.$

If $L=1$, $f$ never magnifies distances (it is 1-Lipschitz, "non-expansive"); if $L<1$ it shrinks gaps (contractive, stable); if $L>1$ it amplifies them (chaotic, sensitive). Putting the two pieces together gives the honest recursion:

$\boxed{E_{t+1}\le L{E}_t+ϵ},E_0=0.$

PICTURE. A bar for $E_t$ is shown; the next bar $E_{t+1}$ is the old bar scaled by $L$ (blue, inherited-and-stretched) with a fresh yellow block ($ϵ$) placed on top. When $L=1$ the blue bar keeps its height and we just stack yellow blocks — the simple case we draw next.

---

Step 5 — Sum the pile: three growth regimes

WHAT. Unroll the recursion $E_{t+1}\le L{E}_t+ϵ$ from $E_0=0$, adding one $ϵ$ per step and stretching everything inherited by $L$:

$E_H\le ϵ(1+L+L^2+\cdots +L^{H-1}).$

That parenthesis is a geometric series — a sum where each term is the previous one times $L$. Here is why it has the tidy closed form: call the sum $S=1+L+\cdots +L^{H-1}$. Multiply by $L$: $LS=L+L^2+\cdots +L^H$. Subtract the two — every middle term cancels, leaving $LS-S=L^H-1$, so $S(L-1)=L^H-1$ and therefore (when $L\ne 1$):

$1+L+\cdots +L^{H-1}=\frac{L^H-1}{L-1},\text{giving}{E}_H\le ϵ\frac{L^H-1}{L-1}.$

Now read off three cases, straight from the sum:

• Non-expansive, $L=1$ (the friendly case). The closed form is $0/0$, so use the raw sum instead: each term is $1$, so it is just $H$ copies: $E_H\le \underset{H\text{ times}}{\underbrace{ϵ+ϵ+\cdots +ϵ}}=Hϵ(\text{linear growth}).$ This is the parent note's "grows roughly like $Hϵ$" — now derived, and the middle case.

• Contractive, $0\le L<1$ (the best case, stable dynamics). Now $L^H\to 0$ as the horizon grows, so the bound stops rising and settles at a ceiling: $E_H\le ϵ\frac{1-L^H}{1-L}\underset{H\to \infty }{\to }\frac{ϵ}{1-L}.$ The error is bounded for all horizons — the inherited gap shrinks faster than fresh error piles on. Stable systems forgive long imagination.

• Expanding, $L>1$ (chaotic/sensitive systems). The power $L^H$ blows up, so $E_H$ grows geometrically — far worse than linear. This is why "or worse" appears in the parent.

Every symbol: $H$ = horizon (steps imagined); $ϵ$ = worst-case per-step error; $L$ = stretch factor. The product $Hϵ$ is the $L=1$ drift; $ϵ/(1-L)$ is the contractive ceiling; $ϵ\frac{L^H-1}{L-1}$ is the exploding one.

WHY. It converts a scary phrase ("errors compound") into three curves you can draw: a bounded curve that flattens ($L<1$), a straight line ($L=1$), and an exploding curve ($L>1$). This is what justifies short horizons except on provably contractive systems.

PICTURE. A graph: horizontal axis is horizon $H$, vertical axis is total error $E_H$. The contractive bound ($L<1$) is a blue curve that flattens to a ceiling $ϵ/(1-L)$; the linear bound $Hϵ$ ($L=1$) is a straight rising pink line; the geometric bound ($L>1$) is a yellow curve bending sharply upward. A dashed line marks the "trust budget"; where a curve crosses it, trust ends.

---

Step 6 — Edge case: $ϵ=0$ (a perfect / given model)

WHAT. What if the model is exact, $ϵ=0$? Then the recursion gives $E_{t+1}\le L\cdot E_t+0$, and starting from $E_0=0$ we get $E_H=0$ for every horizon and every $L$ — even a chaotic $L>1$, because there is no fresh error for $L$ to amplify. The two chains in Step 3 stay glued together forever; imagination equals reality.

WHY. This is exactly the "model given" case from the parent (chess, Go, AlphaZero): the rules are known perfectly, so you may imagine as far as you like — the only hard part left is searching well, not error. Showing this case proves the compounding problem is a property of learning the model ($ϵ>0$), not of planning itself.

PICTURE. The two trajectories overlap into one line — no yellow gaps at all. The error graph is flat on the axis.

Recall Two extreme cases side by side

Perfect model ($ϵ=0$): imagine forever, no drift ::: given-model case (AlphaZero-style), only search is hard — true for any $L$. Learned model ($ϵ>0$): bounded at $ϵ/(1-L)$ if $L<1$, drift $Hϵ$ if $L=1$, geometric if $L>1$ ::: only the last two force small $H$.

---

Step 7 — The fix, drawn: short horizon + re-plan (MPC)

WHAT. Since error grows with $H$ (bounded if $L<1$, linear if $L=1$, geometric if $L>1$), on the non-contractive systems keep $H$ small. Imagine only a few steps, pick the best first action $a_0$, execute just that one action in the real world, observe the true new state, and then imagine again from there. This is Model Predictive Control (MPC): plan a short window, do one step, re-plan.

WHY. Executing one real action resets the gap to zero ($E_0=0$ again) because you land on a genuinely observed state. You never let $H$ grow large, so neither $Hϵ$ nor the geometric $L^H$ term grows large. Re-planning is the mechanism that repeatedly snaps the pink chain back onto the blue chain.

PICTURE. The imagined pink chain runs only a short window (say 3 steps) before the yellow gap would matter; a blue "snap-back" arrow drops the plan back onto the true state, and a fresh short pink window starts. Saw-tooth of small errors instead of one runaway error.

---

Step 8 — Two more real fixes (why they work here)

WHAT. Beyond short horizons, the parent lists two more defences; both attack terms in $E_H\le ϵ\frac{L^H-1}{L-1}$.

• Ensembles (Uncertainty and Ensembles in RL): train several models on the same data. Where they agree, the true $ϵ$ is small; where they disagree, $ϵ$ is effectively large and now visible to us, because the spread of the ensemble's predictions is a measurable stand-in for the invisible true error. A plan that wanders into a high-disagreement region is penalised, so the agent steers away from exactly the places where the $L^H$ term would explode. In short: ensembles turn the unknown $ϵ$ into a quantity the planner can see and avoid.
• Value-aware / latent models (MuZero and Dreamer): don't shrink $H$, shrink what counts as error. Instead of measuring $ϵ$ as "how wrong is the full next state (every pixel)", measure it as "how wrong is the next state in the features that change the reward and the decision". Predicting only decision-relevant quantities makes the per-step $ϵ$ in our bound smaller, because irrelevant pixel noise no longer inflates it. Same $H$, smaller $ϵ$, smaller $E_H$.

WHY. Each is a knob on the same formula $E_H\le ϵ\frac{L^H-1}{L-1}$: MPC lowers $H$; ensembles detect large $ϵ$ so the plan avoids it; latent/value-aware models lower the $ϵ$ that actually matters. Same enemy, three levers.

PICTURE. The error curve from Step 5 with three labelled arrows pushing it down: one arrow shortening $H$ (MPC), one arrow lowering the whole curve via smaller $ϵ$ (latent/value-aware), and a shaded high-$ϵ$ region that the ensemble flags and the plan routes around.

---

The one-picture summary

WHAT. Everything on this page in one frame: a true (blue) trajectory and an imagined (pink) one starting together under the same actions; the yellow gap $ϵ$ per step, stretched by $L$ and summing to $ϵ\frac{L^H-1}{L-1}$ (bounded at $ϵ/(1-L)$ when $L<1$, just $Hϵ$ when $L=1$, exploding when $L>1$); the blue trust-budget line; and the blue snap-back arrow (MPC) that keeps the window short. If $ϵ=0$, the pink line lies on the blue line for any $L$ and no snap-back is ever needed.

Recall Feynman: tell the whole walkthrough in plain words

You're planning a road trip using a hand-drawn map you made yourself. Step 1: reality has a true road (blue) and your map has a drawn road (pink) — they start at your house together. Step 2: your map is a little off at each intersection; the biggest possible miss anywhere is $ϵ$. Step 3: to plan ahead you follow your map's road, not the real one, taking the same turns you'd really take, so from the second turn on you're navigating from a spot the real road never reached. Step 4: each turn adds one more fresh miss ($ϵ$) on top of the old error — and the "detour never longer than straight" rule (triangle inequality) lets us count those two misses separately; the true roads can keep gaps the same ($L=1$), shrink them ($L<1$), or fan them out wider ($L>1$). Step 5: if roads shrink gaps your error hits a ceiling and stops growing; if they keep gaps you're off by about $H$ times $ϵ$; if they fan out the error explodes. Step 6: if your map were perfect ($ϵ=0$) you could dream to the end with zero error no matter what — that's chess, where the rules are exact. Step 7: since your map isn't perfect, only trust it for a few turns, actually drive one real turn (which puts you back on a known real spot and erases all built-up error), then re-plan — that's MPC. Step 8: you can also carry several maps and distrust spots where they disagree (ensembles), or only bother drawing the route-deciding parts of the map (latent models). One enemy — drift that grows like $Hϵ$ (or worse) — several ways to keep it small.

Recall Self-check

Why does imagined error grow at least linearly in the horizon when $L=1$? ::: Because $E_{t+1}\le E_t+ϵ$ with $E_0=0$ adds one fresh $ϵ$ per step, summing to $E_H\le Hϵ$. What tool lets us split the one-step error into "fresh model error" plus "inherited-and-stretched" error? ::: The triangle inequality, $\Vert A-C\Vert \le \Vert A-B\Vert +\Vert B-C\Vert$, with $B=f({\hat{s}}_t,a_t)$. What happens to the accumulated error when the dynamics are contractive ($0\le L<1$)? ::: It is bounded for all horizons and converges to the ceiling $ϵ/(1-L)$, because $L^H\to 0$. What happens to the growth when the true dynamics have $L>1$? ::: The recursion unrolls to $E_H\le ϵ\frac{L^H-1}{L-1}$, which grows geometrically — much faster than linear. What does executing one real action do to the accumulated error? ::: Resets it to zero — you land on a truly observed state, so $E_0=0$ again. That's why MPC re-plans every step. When can you safely imagine over a long horizon? ::: When $ϵ=0$ (a perfect/given model, any $L$), or when the dynamics are contractive ($L<1$, error stays bounded).