Exercises — Model-based RL overview

(a) Model-based. It predicts the next state ($\hat{f}$) and plans inside that prediction — the defining move of MBRL. (b) Model-free. It never builds $\hat{f}$ or $\hat{r}$; it learns values straight from real experience. (c) Both / hybrid — but counts as model-based. Dyna keeps a learned model to amplify data (generate imagined transitions) on top of real ones. Because it uses a model, it is a model-based method, even though the value updates look model-free.

Sample efficiency — how much good behaviour you get per real environment interaction. MBRL spends extra CPU (planning/imagining) to squeeze more learning out of each expensive real step.

WHAT we do: solve the two equations that force each prediction to equal its target. $w_1(3)+w_2(-2)=1,w_1(1)+w_2(4)=5.$ WHY: with two clean, non-redundant samples, ordinary least-squares regression (fitting $\hat{f}$ to the observed $s^{\prime }$ targets, as derived in Exercise 5.1) reduces to solving a linear system exactly. From the second equation $w_1=5-4w_2$. Substitute into the first: $3(5-4w_2)-2w_2=1\Rightarrow 15-14w_2=1\Rightarrow w_2=1.$ Then $w_1=5-4(1)=1$. Answer: $w_1=1,w_2=1$, i.e. $\hat{f}(s,a)=s+a$ — the true dynamics recovered, loss $=0$. The model is now a perfect simulator.

Targets are $\Delta s=a$ for both samples: at $a=-2,\Delta s=1-3=-2$; at $a=4,\Delta s=5-1=4$. So $\hat{g}(s,a)=w_{2}a$ needs $w_2=1$. Recover $s^{\prime }$ by adding $s$ back: $s^{\prime }=s+\hat{g}(s,a)=s+a$. Identical model, but the regression target had smaller magnitude and near-zero mean — easier and more stable to fit. See Uncertainty and Ensembles in RL for why stable residuals also make ensemble spread meaningful.

Predict each next state and its reward:

Highest reward is $0$ at $a_0=-2$ (it lands exactly on $0$). Best action: $a_0=-2$. This is MPC with $H=1$: search inside the model, execute the first (only) action.

WHAT/WHY: each imagined step feeds its (slightly wrong) output into the next, so errors accumulate. This is the parent note's "compounding" enemy.

• $H=20$: end error $\approx 20\times 0.1=2.0$ state units — potentially huge relative to a state near $0$.
• $H=2$: end error $\approx 2\times 0.1=0.2$. Conclusion: the long rollout's final state is untrustworthy; the short one stays tight. This is exactly why MPC re-plans with a short horizon instead of committing a long imagined plan.

Real next state: $s_1=2+(-2)+0.3=0.3$ (not the imagined $0$). The model thought it landed on $0$; reality is off by $0.3$ — the model's bias. Re-planning fix: at the next step MPC observes the real $s_1=0.3$ (not its imagined $0$) and plans afresh. So it corrects the bias instead of blindly following a stale imagined trajectory. Re-planning turns a compounding error into a one-step error each time.

(a) Chess — given. Rules are exact; the hard part is planning/search (huge tree). See MuZero and Dreamer for learning a model even when rules exist, to plan in a compact latent space. (b) Pendulum — learned. You don't know friction/mass exactly; you regress $\hat{f}$ from data. Hard part: model error + compounding. (c) Gridworld with rules — given. The transition probabilities $\hat{P}(s^{\prime }\mid s,a)$ and rewards $\hat{r}$ are known tables; hard part is just planning (e.g. value iteration via the Bellman Equation).

Evaluate each action from $A$:

• $\text{go}$: $\hat{r}+\gamma V^{\ast }(B)=5+0.9(0)=5$.
• $\text{stay}$: $\hat{r}+\gamma V^{\ast }(A)=0+0.9V^{\ast }(A)$.

Take the max. Guess $\text{go}$ is optimal (it grabs reward and ends). If so $V^{\ast }(A)=5$. Check that $\text{stay}$ isn't better: $0.9(5)=4.5<5$. ✅ Consistent. Answer: $V^{\ast }(A)=5$, optimal action $\text{go}$. The whole calculation used only the model's $\hat{r},\hat{P}$ — zero real interaction.

Exhaustive search: $H=2$ with $|A|=3$ gives $3\times 3=9$ sequences. We evaluate all nine (no pruning) — MPC's brute force is honest. With ${\hat{s}}_1=2+a_0$ and ${\hat{s}}_2={\hat{s}}_1+a_1$, the objective is $-{\hat{s}}_1^2+0.5(-{\hat{s}}_2^2)$:

Best total $=0$ at $(a_0,a_1)=(-2,0)$. Execute $a_0=-2$, then re-plan from the real next state. Why the discount matters: $\gamma =0.5$ halves the second term's weight, so landing on $0$ immediately ($a_0=-2$) dominates — the near future counts more.

Total $=10+10k=1010\Rightarrow 10k=1000\Rightarrow k=100$. Risk: if $k$ is huge and the model is imperfect, the flood of imagined transitions drowns the $10$ real ones, so Q-learning converges to the value function of the fantasy model, not reality. Balance $k$ against model accuracy (use ensemble uncertainty to gate imagined data).

Step 1 — one-sample density. For a Gaussian with mean ${\hat{f}}_{\theta }(s,a)$ and covariance ${\sigma }^{2}I$ (the $d\times d$ identity matrix scaled by the noise variance ${\sigma }^2$): $p_{\theta }(s^{\prime }\mid s,a)=\frac{1}{(2\pi {\sigma }^2{)}^{d/2}}\exp (-\frac{\Vert s^{\prime }-{\hat{f}}_{\theta }(s,a){\Vert }^2}{2{\sigma }^2}).$ Step 2 — log it (log turns the product over samples into a sum, and kills the exponential): $\log p_{\theta }(s^{\prime }\mid s,a)=-\frac{1}{2{\sigma }^2}\Vert s^{\prime }-{\hat{f}}_{\theta }(s,a){\Vert }^2-\frac{d}{2}\log (2\pi {\sigma }^2).$ Step 3 — sum over the dataset (independent samples ⇒ log-likelihood is the sum): $\log L(\theta )=-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^N\Vert s_i^{\prime }-{\hat{f}}_{\theta }(s_i,a_i){\Vert }^2-\frac{Nd}{2}\log (2\pi {\sigma }^2).$ Step 4 — drop constants. The second term has no $\theta$; the factor $\frac{1}{2{\sigma }^2}$ is a positive constant. Maximizing $\log L$ over $\theta$ is therefore the same as minimizing $\mathcal{L}(\theta )=\frac{1}{N}{\sum }_{i=1}^N\Vert s_i^{\prime }-{\hat{f}}_{\theta }(s_i,a_i){\Vert }^2.■$ WHY it matters: the squared-error loss isn't arbitrary — it is Gaussian maximum likelihood. Change the noise assumption and the loss changes with it.

Total imagined return $={\sum }_{t=0}^{H-1}{\gamma }^t\hat{r}({\hat{s}}_t,a_t)$. Each term's reward error is at most $\delta$, so by the triangle inequality $\text{error}\le {\sum }_{t=0}^{H-1}{\gamma }^t\delta =\delta \frac{1-{\gamma }^H}{1-\gamma }.$ As $H\to \infty$ (with $0\le \gamma <1$, so ${\gamma }^H\to 0$): $\text{error}\le \frac{\delta }{1-\gamma }.$ Interpretation: even with infinitely long imagination the discounted error stays bounded by $\frac{\delta }{1-\gamma }$ — because discounting silences the far future. This is the mathematical reason short-horizon MPC + discounting keeps compounding error tame. (Numerically with $\delta =0.1,\gamma =0.9$: bound $=1.0$.)

Loop: