Intuition The One Core IdeaA world model is a machine that learns to dream the future : give it where you are now and what you're about to do, and it predicts what you'll see and feel next. Once an agent has this dream-machine, it can rehearse thousands of "what if I did this?" plans inside its own head before touching the real, expensive world.
Before you can read the parent note, you need to earn every symbol it throws at you. This page lists them in build-order: each one is defined in plain words, tied to a picture, and justified by the question it answers. Nothing is used before it is built.
Everything here happens inside one repeating loop. Picture a creature (the agent ) sitting inside a room (the environment ). At each tick of a clock it sees something, does something, and feels a score.
Definition The perception–action loop
The agent is the decision-maker (the brain + body we are training).
The environment is everything else — the world the agent cannot directly control.
At every moment they exchange three things: an observation , an action , and a reward . That's the whole game.
This loop is the skeleton the parent note hangs every equation on. If you see this picture in your head, the rest is just labels.
Definition The time indext t t = 0 , 1 , 2 , 3 , … subscript on a symbol. If we write x t x at time t
Intuition Why we need a clockThe whole point of a world model is predicting the next thing. "Next" only means something if we can say when now is and when next is . So t t + 1 + 1
Read x t + 1 x x t − 1 x before ." That's all a subscript ever does — it points at a moment on the clock. (We'll attach it to real symbols like s a
s o
The state s t t
The observation o t sees of that state through its sensors (e.g. a 64 × 64
The letter s s tate, o o bservation. They are not the same:
Intuition State vs. observation — the keyholeImagine peering through a keyhole into a room. The state is the whole room; the observation is the sliver you can see. Real agents almost never see the full state — they get a limited, noisy peek. This gap is called partial observability , and it's exactly why the parent note builds a compressed internal picture later.
The letter S set of all possible states — the collection of every situation that could ever exist. Small s t t
a A
a t action the agent chooses at time t A all actions available.
The letter a a ction. This is the only lever the agent controls. Everything the agent influences about its future, it influences through a t
Intuition Why actions are the only "handle"The agent can't reach into the environment and set the state directly. It can only nudge — pick an action and see what the world does in response. Planning is nothing more than asking: "which sequence of nudges gives me the best future?"
r r t = r ( s t , a t ) reward : a single number saying "how good was it to do action a t s t
The letter r r eward. The notation r ( s t , a t ) r function that eats a state and an action and returns a number." (More on functions in §6.)
Intuition Why a single number?To compare plans we need to rank them, and you can only rank things that live on one scale. Squashing "how good was this moment" into one number lets us later add up rewards over time and say "this whole future scored 42, that one scored 17 — pick the 42." That sum is the return (§8).
The future is not certain. Push a stack of blocks and it might topple left, might topple right. We need a language for uncertainty.
Definition The probability symbols
P ( x ) probability of outcome x 0 1 P ( s t + 1 ∣ s t , a t ) s t + 1 given ("∣ given ) that we were in s t a t x ∼ P x drawn at random from the distribution P Δ ( X ) X
Intuition Why probabilities and not a single predictionIf the world can branch, predicting one single next state is a lie — you'd always predict the average , which may be a place you never actually go (steer left OR right, but never straight-through-the-wall). A distribution honestly keeps all the branches with their weights. This is the exact reason the parent note reaches for a mixture of Gaussians — many bumps, one per branch.
Common mistake "Given" is not "and"P ( s t + 1 ∣ s t , a t ) not P ( s t + 1 and s t and a t ) ∣ know the right-hand side and only ask about the left. It's "the odds of next, assuming we're standing here doing this."
This distribution P ( s t + 1 ∣ s t , a t ) is the world's true physics. A world model is our learned copy of it (§10).
Definition Function and the mapping arrowA function is a machine: put something in, get something out. We write f ( x ) x f M : ( S , A ) → Δ ( S × R ) M state and an action , and outputs a probability distribution over (next-state, reward) pairs ."
S × R and one real number (the reward)". The × R
Definition The two ways to write the same world model —M p two equivalent notations for a world model, and the parent note uses both:
M : ( S , A ) → Δ ( S × R ) machine view : a function whose output is a whole distribution .p ( s t + 1 , r t ∣ s t , a t ) distribution view : the actual probability of a specific (next-state, reward) pair, given the current state and action.
The lowercase p P density over continuous values. They say the same thing: "feed in where you are and what you do, get out the odds of every possible next-state-and-reward."
Intuition Read the arrow as a factory beltLeft of the arrow = raw material going in. Right of the arrow = product coming out. M p ( s t + 1 , r t ∣ s t , a t )
Definition Markov propertyA process is Markov if the next state depends only on the current state and action — not on the entire history of how you got there.
P ( s t + 1 ∣ s t , a t , s t − 1 , a t − 1 , … ) = P ( s t + 1 ∣ s t , a t )
Intuition Why this makes prediction possibleA chessboard is Markov: to plan your next move you only need the current board, not the order the pieces arrived. If you needed the whole past, prediction would drown in history. The Markov property is the assumption that lets the parent note write dynamics as a clean one-step rule and chain it forward.
This "chaining one step at a time" is why the parent note's giant expectation is nested and recursive — each step only looks one tick back.
Definition MDP — the whole game bundledAn MDP (M arkov D ecision P rocess) is simply the whole loop packaged into one object: a set of states S A r P ( s t + 1 ∣ s t , a t )
Definition Sum, return, horizon
∑ t = 0 H − 1 r t add up the rewards from t = 0 t = H − 1 ∑ H horizon : how many steps into the future we plan.J = ∑ t = 0 H − 1 r t return — the total score of a whole plan.
Intuition Why sum rewards instead of judging each stepA good plan may accept a bad moment now for a great payoff later (crouch before you jump). Judging single steps is greedy and short-sighted. Summing over the horizon H J
Definition Expectation, sample count, and the hat
E [ X ] expected value — the average of X E P ^ [ … ] P ^ K how many random futures we sample to estimate that average — draw K A hat , like P ^ P ^ θ estimate : our learned approximation of the true thing P
Intuition Why average, not best-caseBecause outcomes are random, a plan doesn't produce one return — it produces a spread of possible returns. To fairly score a plan we take its average return, weighting each future by how likely it is. That is exactly what E every future, so we roll out K K
Mnemonic Hat = "my guess"Whenever you see a hat (P ^ my guess of ". Bare P P ^ θ θ
Definition Parameters and updates
θ parameters — all the adjustable knobs (weights) inside a learned model. Training = turning these knobs.x ← x + something update rule : "replace x α learning rate — how big each knob-turn is.∇ θ J gradient — the direction in knob-space that increases J uphill is gradient ascent.
Intuition Why gradients power planningImagine standing on a foggy hillside wanting the summit (max return). The gradient is the arrow "steepest way up" under your feet. Take a small step α model learns to predict and, when the model is differentiable, how plans get optimized — the parent's chain-rule gradient is exactly this arrow traced back through time.
The parent note assembles three pre-built components. You don't need their internals yet — just what job each does.
π Q
A policy π ( a ∣ s ) strategy : given a state, it says which action (or the odds of each action) to take. Think "the rulebook the agent follows."
Q ( s , a ) value : a learned estimate of "how much total reward do I expect if I take action a s Reinforcement-Learning-Basics ; here you only need to recognize them.
z t h t
z t latent code : a short list of numbers (e.g. 32 64 × 64 h t hidden state : its running memory, updated every tick, so it knows velocity and trend , not just the current frozen frame.
Intuition Why compress before predictingPredicting the next million pixels is hopeless; predicting the next 32-number summary is easy. Compress first (that's the VAE), then do all the hard dreaming in the small clean space. The tiny controller can then steer using just this summary.
state s and observation o
probability P and sampling
sum and return J over horizon H
expectation E, K samples, hat approx
parameters theta and gradient
World Model M or p of next state and reward
policy pi and value Q and MDP
Each box is a symbol you now own; the arrows show which idea must exist before the next can be spoken. The whole tree drains into one node — the world model, written either as the machine M : ( S , A ) → Δ ( S × R ) p ( s t + 1 , r t ∣ s t , a t )
Cover the right side and answer aloud. If any one stumps you, re-read its section before opening the parent note.
What does the subscript in s t A specific moment on the clock — the state at tick t
What is the difference between s t o t s t o t
Read P ( s t + 1 ∣ s t , a t ) The probability of landing in the next state, given we're in the current state and take this action.
What does the bar ∣ It means "given / assuming"; it does NOT mean "and".
What does x ∼ P x P
Why use a probability distribution instead of predicting one next state? The future can branch; a single prediction collapses to a wrong average, a distribution keeps every branch with its weight.
What are the two equivalent ways to write a world model? The machine view M : ( S , A ) → Δ ( S × R ) p ( s t + 1 , r t ∣ s t , a t )
State the Markov property in one sentence. The next state depends only on the current state and action, not on the full history.
What does "MDP" bundle together? The states S A r
What is the return J J H
What does K E The number of random futures (trajectories) we sample and average to approximate the true expectation.
What does the hat in P ^ θ It is our learned estimate of the true (unknown) rule P θ
What does the gradient ∇ θ J The direction in knob-space that increases the return J
What is a policy π ( a ∣ s ) Q ( s , a ) π Q
Why compress an image into a latent z t Predicting a short code is tractable; predicting a million raw pixels is not — dream in the small clean space.
What single job does the RNN hidden state h t Carries a running memory of the past so the model knows trends like velocity, not just one frozen frame.
You are now equipped to read the parent topic without hitting an undefined symbol.
