Intuition The one core ideaExperience replay is a notebook of past moments the agent keeps, and every time it studies it flips to a few random pages instead of only re-reading the very last thing that happened. To read the parent note you must first learn what one "moment" is, what it means to predict how good a moment is , and why random review beats in-order review — this page builds all of that from nothing.
This is the ground-floor page for Experience replay . Nothing here assumes you have seen reinforcement learning before. Every letter and squiggle the parent note uses is defined below, in an order where each idea rests on the one above it.
Picture a little robot playing a game. At every tick of the clock, four things happen. Let us name them.
s state s t Picture: the frozen screenshot.
Why the topic needs it: the buffer stores states so it can re-ask "what should I have done here ?" later.
a action a Picture: the arrow leaving the current screenshot in the figure.
Why the topic needs it: an experience is worthless unless we remember what we chose , because that choice caused the outcome.
r reward r + 1 0 − 1 Picture: the amber number popping up after the action.
Why the topic needs it: reward is the feedback signal . Learning means turning past rewards into better future choices.
s ′ s ′ s s Picture: the second screenshot the arrow points to.
Why the topic needs it: to judge an action we need to know what came after it .
Bundle those four into one packet:
e t = ( s t , a t , r t , s t + 1 ) transition (also called an experience ) is the tuple — an ordered bundle written inside round brackets ( )
The subscript t t clock stamp : e 1 e 2
s t + 1 s ′ t t + 1 Picture: the whole boxed row in the figure.
Why the topic needs it: this packet is exactly one page of the notebook . The buffer is a stack of these.
s ′ s t + 1 Why it feels right: they look different. Why it's wrong: they are two spellings of the same idea — "the state that comes next". The parent note uses both. Fix: read every ′
N D N count : how many transitions we are holding. The fancy letter D Data ) is the name of the whole buffer : D = { e 1 , e 2 , … , e N } { } Picture: a filing cabinet with N Why the topic needs it: "FIFO, capacity N N + 1 F irst I n F irst O ut).
Why does "almost identical neighbours" matter? Because 3 pixels of movement between s t s t + 1
Definition i.i.d. — "independent and identically distributed"A pile of samples is i.i.d. when (a) picking one tells you nothing about the next (independent), and (b) they all come from the same source (identically distributed).
Picture (left panel): the live stream is a chain of near-copies — knowing s t s t + 1 not independent. Right panel: random draws from a full cabinet are scattered and unrelated — close to i.i.d.
Why the topic needs it: the maths that trains neural nets assumes i.i.d. samples. Correlated (chained) samples break that assumption, so we shuffle with U ( D )
∣ B ∣ 1 ∑ ∑ "add all of these up" . The bars ∣ B ∣ "how many things are in B (its size). So ∣ B ∣ 1 ∑ e ∈ B ( ⋯ ) "add up one number per transition, then divide by how many — the plain average" .
Picture: stacking the batch's numbers and sharing them equally.
Why the topic needs it: we don't update from one page; we average the lesson over the whole handful B
Now the heart of it: what is the network actually learning ?
Definition The value functionQ ( s , a ) Q ( s , a ) predicted score : "if I am in state s a Picture: a scorecard that, shown a screenshot and an arrow, prints a number.
Why the topic needs it: the agent picks the action with the biggest Q Q
θ Q θ θ all the tuning knobs inside the neural network — its weights. Writing Q θ θ Picture: a machine with thousands of little dials; θ the whole set of dial positions .
Why the topic needs it: "learning" = nudging θ Q θ Deep Q-Networks (DQN) .
γ max a ′
γ 0 1 discount factor : future reward counts a little less than reward now. γ = 0.9 90% max a ′ Q ( s ′ , a ′ ) look at the next state s ′ a ′ The prime on a ′ next action".
Picture: at the next screen, fan out all buttons, read each score, circle the highest (amber ring in the figure).
Why the topic needs it: this "best future value, shrunk by γ
Now we can state the equation the whole target is built on.
Intuition Why this equation lets usbootstrap
We don't know Q ∗ Q ∗ Q ∗ next moment. So we can cheat : plug in our current estimate for the future term and treat the whole right-hand side as a temporary answer key. That reused-your-own-guess move is called bootstrapping , and it turns the equation above into the target below.
y = r + γ max a ′ Q θ − ( s ′ , a ′ ) y "should-have-been" answer — the score we wish Q θ ( s , a ) Q ∗
θ − frozen, slightly-old copy of the knobs, used only to compute y Target Networks .
Picture: the answer key we compare our guess against.
Why the topic needs it: this bootstrapped y
Definition Terminal states and the( 1 − d ) s ′ terminal — the episode is over (you died, or the level ended). There is no next move , so there is no future value to add: max a ′ Q ( s ′ , a ′ ) 0 s ′
We record a done flag d d = 1 s ′ d = 0
The target becomes y = r + γ ( 1 − d ) max a ′ Q θ − ( s ′ , a ′ ) d = 1 ( 1 − d ) = 0 erases the bootstrap term, leaving just y = r Picture: a light-switch on the future term — terminal flips it off, so only the final reward remains.
Why the topic needs it: forget this and the network hallucinates future reward after the game ended, poisoning every replay that touches a terminal transition.
δ = y − Q θ ( s , a ) δ guess-versus-answer-key gap : the target minus what we predicted. Big ∣ δ ∣ Picture: the length of the red mismatch bar between the two numbers.
Why the topic needs it: δ both how much we correct the network and (in the upgrade) how interesting a transition is to replay.
ℓ = ( y − Q θ ( s , a ) ) 2 = δ 2 Loss is the gap, squared . Squaring does two jobs: it makes every gap positive (a miss is a miss whether too high or too low), and it punishes big misses much harder than small ones.
Picture: the area of a square whose side is the mismatch bar — doubling the error quadruples the pain.
Why the topic needs it: training = turning knobs θ
transition e = s a r s prime
buffer D of N transitions
Bellman optimality equation
discount gamma and max over a prime
bootstrapped target y with done flag d
Read it top-down: the four raw items form a transition, transitions fill the buffer, random sampling makes an almost-i.i.d. batch; meanwhile the value network and the discounted Bellman rule build a target (with the terminal ( 1 − d ) is experience replay.
Definition Off-policy vs on-policyOn-policy learning insists you only study data made by your current way of playing. Off-policy learning is happy to study data made by an older, different way of playing.
Picture: on-policy = only trust today's diary; off-policy = old diary pages still teach you.
Why the topic needs it: the buffer is full of old-policy pages , so replay only works cleanly for off-policy methods. This is the Off-policy vs On-policy distinction, used in DDPG and Soft Actor-Critic .
Definition Importance sampling (preview)If you deliberately pick some pages more often than others, a plain average is tilted . Importance sampling re-weights each picked item by its pick-probability 1 Picture: a see-saw rebalanced by putting lighter weights on the over-picked side.
Why the topic needs it: the Prioritized replay upgrade picks high-δ Importance Sampling .
Cover the right side; can you say each before revealing?
What the four items of a transition are state s a r s ′ s t + 1
What D N D N
What B ∼ U ( D ) "take a mini-batch B D
Why we want i.i.d. samples neural-net training assumes samples are independent and from one source; correlated live streams break that
What ∑ ∣ B ∣ 1 ∑ ∑ ∣ B ∣ 1
What Q θ ( s , a ) the expected total (discounted) future reward for doing a s θ
What the Bellman optimality equation says Q ∗ ( s , a ) = E s ′ [ r + γ max a ′ Q ∗ ( s ′ , a ′ ) ]
What γ max a ′ γ max a ′
What y θ − y θ −
How a terminal next state changes y with done flag d = 1 ( 1 − d ) = 0 y = r
What δ δ = y − Q θ ( s , a ) δ 2
Why replay suits off-policy but not on-policy the buffer holds old-policy data; only off-policy methods can learn from data made by a different/older policy
