Worked examples — Model-based RL overview

This page is the "drill ground" for Model-based RL overview. The parent note built the ideas: learn a model $\hat{f}$ (a cheap simulator), then plan inside it. Here we hit every kind of case the topic can throw at you, one worked example per case, so you never meet a scenario you have not seen solved.

Before we start, some symbols we will lean on constantly, defined in plain words right here so you never have to flip back:

---

The scenario matrix

Every problem in model-based RL falls into one of these cells. The examples below fill all of them.

Cell	What makes it special	Example
A. Clean linear fit	Data exactly obeys a linear rule, noise-free	Ex 1
B. Noisy fit	Data has noise; least-squares averages it	Ex 2
C. Degenerate data	Not enough / redundant data → weights undetermined	Ex 3
D. Planning, deterministic	Search action sequences in a known/learned model	Ex 4
E. Planning, stochastic	Next state is a distribution; use expected value (Bellman)	Ex 5
F. Limiting case $\gamma$	Behaviour as $\gamma \to 0$ (greedy) vs $\gamma \to 1$ (far-sighted)	Ex 6
G. Compounding error	A wrong model, error grows with horizon $H$	Ex 7
H. Real-world word problem	Robot/thermostat framed in English	Ex 8
I. Exam twist (Dyna count)	Sample-efficiency arithmetic: real vs imagined data	Ex 9

Prerequisites we reuse: Markov Decision Process, Bellman Equation, Model Predictive Control (MPC), Dyna Architecture, Sample Efficiency, Uncertainty and Ensembles in RL.

---

Cell A — Clean linear fit

1. Write one equation per sample. Sample 1: $w_1(2)+w_2(-1)=1$. Sample 2: $w_1(0)+w_2(2)=2$. Why this step? The model must reproduce the observed $s^{\prime }$ for the inputs it saw — that is literally what "fit" means.
2. Solve the second equation. $2w_2=2\Rightarrow w_2=1$. Why? It has only one unknown, so it pins $w_2$ immediately.
3. Back-substitute into the first. $2w_1-1=1\Rightarrow w_1=1$. Why? Now both weights are known.
4. Loss. Predictions: $1(2)+1(-1)=1$ ✓ and $1(0)+1(2)=2$ ✓. Using the loss above with $N=2$: $\mathcal{L}=\frac{1}{2}(0^2+0^2)=0$. Why? Squared error $=0$ means a perfect simulator — the model reproduces both transitions exactly.

Verify: $\hat{f}(s,a)=s+a$ matches the true rule $s_{t+1}=s_t+a_t$ everywhere. Plan to minimise $s^2$ ⇒ pick $a=-s$, driving state to $0$. Consistent. ✓

---

Cell B — Noisy fit

1. Write the loss. With $N=3$ samples, $\mathcal{L}(c)=\frac{1}{3}[(0.9-c{)}^2+(1.2-c{)}^2+(0.6-c{)}^2]$. Why this step? Same least-squares loss as above, now with the constant prediction $c$ standing in for $\hat{f}$.
2. Minimise: set derivative to zero. $\frac{d\mathcal{L}}{dc}=\frac{-2}{3}[(0.9-c)+(1.2-c)+(0.6-c)]=0$. Why? The minimum of a smooth bowl-shaped (parabola) loss is where its slope is $0$. That is why we use the derivative here and not, say, sorting — squared error is smooth and convex, so calculus finds the exact bottom.
3. Solve. $2.7-3c=0\Rightarrow c=0.9$. Why? The mean of the readings. Least squares → the mean is the noise-averaging estimate.

Verify: Mean $=(0.9+1.2+0.6)/3=0.9$. True value $1.0$; our estimate $0.9$ is close, the bias just reflects this small noisy sample. Loss at $c=0.9$ is $\frac{1}{3}(0+0.09+0.09)=0.06>0$ — nonzero because data is noisy, unlike Ex 1. ✓

---

Cell C — Degenerate data

1. Write the equation. $w_1(1)+w_2(1)=2$, i.e. $w_1+w_2=2$. Why this step? Each sample gives one constraint; we have one sample.
2. Count freedom. Two unknowns, one equation ⇒ infinitely many solutions: $(w_1,w_2)=(0,2),(1,1),(2,0),\dots$ all fit with zero loss. Why? This is the degenerate/underdetermined case. Zero training loss does not mean a correct model — a warning the parent's "model errors" theme cares about.
3. Consequence for planning. $(0,2)$ predicts $s^{\prime }=2a$ (ignores state); $(1,1)$ predicts $s^{\prime }=s+a$. On a new input $(s=5,a=0)$ they disagree: $0$ vs $5$. Why? Degenerate fits generalise unpredictably — this is exactly where an ensemble flags "I don't know."

Verify: Plug $(s=5,a=0)$: model $(0,2)\Rightarrow 0$, model $(1,1)\Rightarrow 5$. Both have zero training loss yet disagree by $5$. Underdetermined confirmed. ✓

---

Cell D — Deterministic planning (MPC)

1. Read the rollout tree in Figure s01. The single cyan dot on the left is $s_0=2$. Each branch is one candidate first action $a_0\in \{-2,-1,0\}$, moving to a step-1 state; each branch then continues to a step-2 state. Every full left-to-right path is one imagined trajectory $s_0\to s_1\to s_2$. Why this step? Planning = imagining futures with $\hat{f}$, paying zero real steps. The tree lays out all futures we are comparing at once.
2. Follow the amber branch in Figure s01: sequence $(-2,0)$. This is the thick amber path that drops straight onto the dashed line $s=0$ and stays there. Numerically ${\hat{s}}_1=2-2=0$, ${\hat{s}}_2=0+0=0$. Cost $={\hat{s}}_1^2+{\hat{s}}_2^2=0+0=0$. Why? We sum the per-step costs; $\gamma =1$ so nothing is shrunk. The amber path touches the target line at both steps ⇒ zero cost, which is why the figure highlights it.
3. Follow a cyan branch in Figure s01: sequence $(-1,-1)$. ${\hat{s}}_1=1$ (one unit above the dashed line), ${\hat{s}}_2=0$. Cost $=1+0=1$ — the path reaches target only on the second step, so it pays $1$ on the way. Why? A slower descent spends a step away from the target, so it accumulates cost the amber path avoided.
4. Compare all paths. The amber path is the only one sitting on the dashed target line at both steps, so it has the minimum cost $0$. Why? MPC picks the lowest-cost imagined trajectory — visually, the path hugging the target line longest.
5. Execute only $a_0=-2$ (the first edge of the amber path), then re-plan from the real next state. Why re-plan? To correct compounding model error (see Ex 7) — the core MPC discipline.

Verify: With true dynamics $s^{\prime }=s+a$: $2+(-2)=0$, then $0+0=0$. Real cost $0$, matching the imagined cost. Model was exact so plan is exact. ✓

---

Cell E — Stochastic planning (Bellman with a distribution)

1. Assemble the backup rule. Because the outcome is a distribution, we cannot use one $s^{\prime }$; we average the landing values weighted by their probabilities. This gives $Q(s,a)=\underset{\text{reward now}}{\underbrace{\hat{r}(s,a)}}+\gamma \underset{\text{expected value of where we land}}{\underbrace{[\hat{P}(+1)V(+1)+\hat{P}(-1)V(-1)]}}.$ Why this step? When $s^{\prime }$ is chance (not our choice), the right tool is the expectation — a probability-weighted mean — not a max over outcomes. A max would pretend we get to pick the coin's result.
2. Plug numbers. $Q=0+0.9[0.5(10)+0.5(2)]=0.9[5+1]=0.9\times 6$. Why? Each landing value multiplied by its probability, then discounted by $\gamma$.
3. Compute. $Q=5.4$.

Verify: Expected next value $=0.5(10)+0.5(2)=6$; times $\gamma =0.9$ gives $5.4$. It sits correctly between the pure-outcome values $\gamma \cdot 2=1.8$ and $\gamma \cdot 10=9$. ✓

---

Cell F — Limiting cases of the discount $\gamma$

1. Discounted return formula: $G=r_0+\gamma r_1$. Why this step? Return sums rewards, each future one shrunk by ${\gamma }^t$; here $r_0$ is step-0 reward, $r_1$ step-1 reward.
2. $\gamma =0$: $G_P=5+0\cdot 0=5$, $G_Q=0+0\cdot 8=0$. P wins. Why? $\gamma =0$ kills all future terms — the agent is purely greedy, only $r_0$ matters.
3. $\gamma =1$: $G_P=5+1\cdot 0=5$, $G_Q=0+1\cdot 8=8$. Q wins. Why? $\gamma =1$ counts the future fully — patience pays.
4. Break-even: set $G_P=G_Q$, i.e. $5=8\gamma \Rightarrow \gamma =5/8=0.625$. Why? Below $0.625$ prefer P, above prefer Q — the discount is the knob for near-vs-far trade-offs.

Verify: At $\gamma =0.625$: $G_Q=0.625\times 8=5=G_P$. Tie confirmed. At $\gamma =0.7$: $G_Q=5.6>5$, Q wins as claimed. ✓

---

Cell G — Compounding model error

1. Trace the true rollout (cyan flat line in Figure s02). With $a_t=0$, $s_t=0$ for all $t$ — nothing moves, so the cyan line sits flat on $0$. Why this step? Establish ground truth to measure error against; the flat cyan line is the "reality" reference.
2. Trace the imagined rollout (amber rising line in Figure s02). ${\hat{s}}_{t+1}={\hat{s}}_t+0.1$, so ${\hat{s}}_H=0.1H$ — the amber line climbs by $ϵ=0.1$ every step, pulling away from the flat cyan line. Why? Each step adds the same bias; the errors accumulate, they do not cancel.
3. Read the error gap. In Figure s02, the white double-arrows at $H=1,3,10$ measure the vertical gap between amber and cyan: $0.1$, $0.3$, $1.0$. Formally $|{\hat{s}}_H-s_H|=ϵH=0.1H$. Why? Because the same $ϵ$ is added every step and never cancels, the total drift is just $ϵ$ summed $H$ times — this is why long horizons are dangerous even with a nearly-good model.
4. State the growth law. The error grows linearly in $H$: $\text{error}=ϵH$. This is the parent's "$Hϵ$" rule — small per-step $ϵ$, but long $H$ magnifies it, as the widening arrows show. Why? It names the pattern so you can predict drift for any horizon without re-simulating.
5. Fix. Keep $H$ short and re-plan (MPC, Ex 4), or use an ensemble to stop trusting the model where it disagrees. Why? Short horizons cap $ϵH$; re-planning resets the imagined state to the true one each cycle.

Verify: $0.1\times 1=0.1$, $0.1\times 3=0.3$, $0.1\times 10=1.0$. Error grows linearly in $H$ exactly as predicted. ✓

---

Cell H — Real-world word problem

1. Net change per minute $=a-0.5$. Why this step? The leak is part of the learned dynamics — the model already includes it, so we plan against reality, not a fantasy where heat is free.
2. Try $a=(2,2)$: ${\hat{T}}_1=20+2-0.5=21.5$, ${\hat{T}}_2=21.5+2-0.5=23.0$. Cost $=(21.5-22{)}^2+(23-22{)}^2=0.25+1.0=1.25$. Why? Evaluate this candidate plan fully inside the model before committing anything.
3. Try $a=(2,1)$: ${\hat{T}}_1=21.5$, ${\hat{T}}_2=21.5+1-0.5=22.0$. Cost $=(21.5-22{)}^2+(22-22{)}^2=0.25+0=0.25$. ✅ Why? This plan lands exactly on target at step 2, so its second-step penalty is zero.
4. Try $a=(2,0)$: ${\hat{T}}_1=21.5$, ${\hat{T}}_2=21.5-0.5=21.0$. Cost $=(21.5-22{)}^2+(21-22{)}^2=0.25+1.0=1.25$. Why compare? MPC picks the lowest-cost imagined trajectory; $(2,1)$ wins.
5. Execute $a_0=2$, re-measure the real temperature, re-plan. Why? Only the first action is committed; re-measuring corrects any leak-model error next cycle.

Verify: $(2,1)$ cost $=0.25<1.25$; final temp $22.0^{\circ }$ hits target. Units: temperature squared error (${}^{\circ }{\text{C}}^2$), consistent throughout. ✓

---

Cell I — Exam twist: Dyna sample-efficiency arithmetic

1. Update budget stays fixed: $50,000$ update-transitions needed. Why this step? Convergence depends on total learning signal, whether real or imagined.
2. Ratio real:total $=1:10$. Why? Each real transition spawns $9$ imagined ones ⇒ $10$ total per real step.
3. Real interactions $=50,000/10=5,000$. Why? We divide the total updates by updates-per-real-step. This is the sample-efficiency payoff — imagined data amplifies scarce real data.
4. Speed-up in real data $=50,000/5,000=10\times$. Why? Comparing real steps needed with vs without the model gives the amplification factor.

Verify: $5,000\times 10=50,000$ updates. Real interactions $5,000$ vs $50,000$ ⇒ $10\times$ fewer real steps. ✓

---

Recall Which cell is which — quick self-test

Ex 1 hits cell ::: A (clean linear fit, zero loss) Ex 3 hits cell ::: C (degenerate / underdetermined data) Ex 5 hits cell ::: E (stochastic planning via expectation) Ex 7 hits cell ::: G (compounding model error, grows like $Hϵ$) Ex 9 hits cell ::: I (Dyna sample-efficiency arithmetic)