Intuition The one core ideaModel-based RL is the strategy of first building a little copy of the world inside your head, then rehearsing plans in that copy before acting for real . Every symbol on the parent page (s a r f ^ γ V ∗ P ^
This page assumes you know nothing . We name each symbol, draw the picture it stands for, and say why the topic cannot live without it. Read top to bottom — every symbol is earned before it is used again.
Before any symbol, picture a game.
There is a you (the agent ) and a world (the environment ). Time ticks in discrete steps t = 0 , 1 , 2 , …
s state s
Picture: the frozen screen of the game right now (positions, velocities, whose turn it is).Why the topic needs it: the model must take a starting snapshot and predict the next snapshot. Without a name for "the current situation," there is nothing to feed in.The subscript s t t s 0 s 1
H a 0 : H horizon H how many steps ahead we look/plan — a whole number like 2 10 50
Picture: how far down the road your headlights reach; beyond H Why: we can't imagine forever, so we pick a finite look-ahead H H H A whole plan is a list a 0 : H = ( a 0 , a 1 , … , a H − 1 ) 0 H H
s ′ Why it feels right: it has a different name (s s ′ The fix: s ′ value of s
The parent page writes f ^ P ^ r ^ s ^
x ^ hat over a symbol means "our guess of it," built from data or from a model — not the true thing.
Picture: the world has a real rulebook; the hat is our pencil-sketch copy of that rulebook.Why: in model-based RL the whole point is that we don't know the real dynamics; we approximate them. f ^ f s ^ t imagined state produced by rolling our model forward, versus a real s t
Look at the two columns above: the left (true, no hats) is the real world we pay to touch; the right (hatted) is the cheap imagined copy we can run for free. Model-based RL lives in the right column and only occasionally checks the left.
f ( inputs ) function is a machine: put something in, get one thing out. In our topic the inputs are always a state and an action , so we write f ^ ( s , a ) f ^ s a ( s , a )
Picture: a vending machine — feed it the current situation s a Why: the deterministic model s ′ = f ^ ( s , a ) s , a
Definition The transitionP ^ ( s ′ ∣ s , a ) random : the same button doesn't always give the same result (dice, noisy sensors). P ^ ( s ′ ∣ s , a ) probability — a number between 0 1 the chance of landing in s ′ s a ."
Picture: instead of one arrow to one next-screen, a spray of arrows with thicknesses = how likely each outcome is; the thicknesses sum to 1 Why: deterministic f ^ 1 0 P ^
∣ P ( s ′ ∣ s , a ) "given" / "conditioned on."
Picture: "the odds of rain, given the sky is grey" — the stuff after the bar is what you already know.Why: the next state's odds depend on the current state and action, so we must state what we're conditioning on.
∑ sigma , ∑ "add up a list."
∑ t = 0 H − 1 x t x 0 , x 1 , … , x H − 1 H Picture: dropping coins one per step into a jar and reading the total.Why: the agent doesn't care about one reward; it cares about the total reward over a trajectory, which is a sum.
γ γ gamma ) is a number between 0 1 shrinks future rewards : a reward t γ t
Picture: a 10 n o t e t o d a y i s w or t hm or e t hana p r o mi se d γ t Why γ Two reasons: (1) it makes an infinitely long sum stay finite (a geometric series), and (2) it encodes "sooner is better," which matches real preferences. If γ = 0 γ → 1
V ∗ ( s ) V ∗ ( s ) best possible total discounted reward you can get starting from state s , if you play perfectly forever after.
Picture: stand on a square of a board game; V ∗ ( s ) Why the star ∗ The star means optimal — the value under the best policy, not some random one. The parent's Bellman equation is a rule that this best-value tag must obey.
max arg max a [ ⋅ ] "the largest value as we try every action a (The related arg max which action gives it.)
Picture: try each button, read the resulting number, keep the biggest.Why: acting optimally = at each state, picking the action with the highest "reward-now-plus-future-value." That "pick the best" step is exactly max a
Now every symbol in the parent's central equation is earned. We can read it in plain English:
V ∗ ( s ) = max a [ score now r ^ ( s , a ) + γ average future worth ∑ s ′ P ^ ( s ′ ∣ s , a ) V ∗ ( s ′ ) ]
Intuition Say it out loud"The best worth of being here = choose the action (max a the score I get now (r ^ plus a shrunk (γ average over all places I might land (∑ s ′ P ^ how good that place is (V ∗ ( s ′ ) hatted , we compute this entirely inside our head-copy of the world — that is the whole payoff of model-based RL.
Dynamics model s prime = f hat of s a
Next state s prime output
Transition P hat given s a
Read it as: states and actions feed the dynamics model , which outputs the next state ; rewards, discount, and sums build the return ; transition probabilities and max build the optimal value ; both flow into planning , which is the beating heart of the Model-based RL overview . The whole apparatus is a formalization of the Markov Decision Process , solved with the Bellman Equation ; the "imagine cheaply" payoff is Sample Efficiency , the loop that reuses imagined data is the Dyna Architecture , the re-plan-every-step tactic is Model Predictive Control (MPC) , and coping with wrong hats leads to Uncertainty and Ensembles in RL and value-aware models like MuZero and Dreamer . Contrast the whole thing with Model-free RL , which skips the model entirely.
Self-test: can you say each aloud before revealing?
What is a state s A complete snapshot of the world at one moment — everything needed to decide the next action.
What is an action a The choice/button the agent picks from a state; it is why the world changes.
What is the horizon H How many steps ahead we plan/look; a plan a 0 : H H a 0 , … , a H − 1
What does s ′ s t + 1 The state one tick later; s ′ s t + 1
What is a reward r A single number scoring one step; it defines what "good" means.
What does a hat, like f ^ Our learned/estimated guess of a quantity, not the true one — a pencil-sketch copy of the world's real rule.
What are the inputs to f ^ A state and an action, the pair ( s , a )
What does P ^ ( s ′ ∣ s , a ) The probability of landing in s ′ s a
What does ∑ s ′ Every possible next state s ′ P ^
Why use a discount γ It keeps infinite sums finite and encodes "sooner rewards are worth more."
What does γ t t Shrinks it: multiplies its face value by γ t
What is V ∗ ( s ) The best possible total discounted reward achievable starting from state s
What does max a [ ⋅ ] Returns the largest value obtained by trying every action a
In one sentence, what does the Bellman line say? Best worth here = pick the action maximizing (reward now + discounted average worth of where you land).
