Experience replay: Worked examples — AI-ML deep dive

Everything below reuses only symbols the parent already earned. Let us restate the three we lean on hardest, in plain words, before any arithmetic:

#	Cell class	Distinguishing input	Filled by
A	Normal non-terminal transition, $δ > 0$	$d = 0$ , under-estimate	Ex 1
B	Terminal transition, $δ < 0$	$d = 1$ , over-estimate	Ex 2
C	Full mini-batch loss + gradient sign	mixed batch	Ex 3
D	PER probabilities, general $α$	$0 < α < 1$	Ex 4
E	Degenerate knob $α = 0$	uniform limit	Ex 5
F	Degenerate knob $α \to 1$	greedy limit	Ex 5
G	Importance-sampling weights + normalisation	$β \in (0, 1)$	Ex 6
H	Limiting behaviour of buffer size on correlation	$N$ small vs large	Ex 7
I	Real-world word problem	end-to-end	Ex 8
J	Exam twist: off-policy validity	on-policy trap	Ex 9

Worked examples

Worked example Example 1 — Cell A: normal non-terminal, positive error

A sampled transition: reward $r = 1$ , not terminal ( $d = 0$ ), discount $γ = 0.9$ , and the target network says $max_{a^{'}} Q_{θ^{-}} (s^{'}, a^{'}) = 5$ . Your current estimate is $Q_{θ} (s, a) = 4$ . Find $y$ and $δ$ .

Forecast: Will $δ$ be positive or negative? (Guess before reading — is $4$ too low or too high?)

Compute the target. $y = r + γ (1 - d) max_{a^{'}} Q_{θ^{-}} (s^{'}, a^{'}) = 1 + 0.9 \cdot 1 \cdot 5 = 5.5$ . Why this step? The target injects the discounted future value — this is the bootstrap from the Bellman optimality equation. Because $d = 0$ , the $(1 - d) = 1$ factor keeps the future term alive.
Compute the TD error. $δ = y - Q_{θ} (s, a) = 5.5 - 4 = 1.5$ . Why this step? $δ$ is the learning signal — it is literally the "temporal difference" of Temporal-Difference Learning. Positive means the network under-valued the move.

Verify: $δ > 0$ matched the forecast (estimate $4 < 5.5$ ). Units: everything is in units of value (expected return), consistent. Gradient descent will now raise $Q_{θ} (s, a)$ toward $5.5$ .

Worked example Example 3 — Cell C: mini-batch loss and gradient direction

Mini-batch $B = {e_{2}, e_{4}}$ from the two examples above: $δ_{2} = 1.5$ , $δ_{4} = - 0.3$ . Compute the batch loss.

Forecast: Does the sign of $δ$ matter for the loss? Guess, then check.

Square each error. $δ_{2}^{2} = 2.25$ , $δ_{4}^{2} = 0.09$ . Why this step? We minimise the squared TD error; squaring makes the loss depend only on the magnitude of the mistake — a fully symmetric penalty for over- and under-estimating.
Average over the batch. $L = \frac{1}{2} (2.25 + 0.09) = \frac{1}{2} (2.34) = 1.17$ . Why this step? The Monte-Carlo estimate of $\nabla_{θ} L$ needs the mean over the batch (dividing by $∣ B ∣ = 2$ ), not the sum — otherwise your step size secretly scales with batch size.
Gradient direction (sign). $\partial ℓ / \partial Q_{θ} = - 2 δ$ . For $e_{2}$ : $- 2 (1.5) < 0$ ⇒ pushes $Q_{θ}$ up. For $e_{4}$ : $- 2 (- 0.3) > 0$ ⇒ pushes $Q_{θ}$ down. Why this step? This confirms the network moves toward $y$ in both cases — exactly what "shrink $δ$ " means.

Verify: $\frac{1}{2} (1. 5^{2} + 0. 3^{2}) = 1.17$ . Since squares are non-negative, $L \geq 0$ always. ✓

Worked example Example 4 — Cell D: PER probabilities, general

$α$ Four buffer priorities from errors $∣ δ ∣ = (1.5, 0.3, 0.3, 0.9)$ , with $ϵ = 0.01$ and exponent $α = 0.6$ . Find each sampling probability $P (i)$ .

Forecast: The biggest error is $1.5$ vs the smallest $0.31$ — about $5 \times$ larger. Will it be sampled $5 \times$ more often, or less?

Raw priorities. $p = ∣ δ ∣ + ϵ = (1.51, 0.31, 0.31, 0.91)$ . Why this step? $ϵ$ guarantees even a perfectly-predicted transition ( $δ = 0$ ) keeps a nonzero chance — no data is ever frozen out.
Raise to $α$ . $p^{0.6} = (1.5 1^{0.6}, 0.3 1^{0.6}, 0.3 1^{0.6}, 0.9 1^{0.6}) \approx (1.286, 0.487, 0.487, 0.945)$ . Why this step? $α$ softens the priorities. At $α < 1$ the ratio between big and small shrinks — so PER is only partly greedy. This is why $1.5$ won't get its full $5 \times$ edge.
Normalise. Sum $\approx 1.286 + 0.487 + 0.487 + 0.945 = 3.205$ . Divide: $P \approx (0.401, 0.152, 0.152, 0.295)$ . Why this step? Probabilities must sum to 1; dividing by the total turns raw scores into a distribution over which transition to draw.

Verify: $\sum P = 1$ (check). The top error is sampled $0.401/0.152 \approx 2.6 \times$ more than a low one — less than the raw $\approx 5 \times$ , exactly because $α = 0.6$ softened it. Forecast confirmed. ✓

Worked example Example 5 — Cells E & F: the two degenerate knob settings

Same priorities $p = (1.51, 0.31, 0.31, 0.91)$ . Evaluate $P (i)$ at the extremes $α = 0$ and $α \to 1$ .

Forecast: Which $α$ makes PER identical to plain uniform replay?

$α = 0$ : every $p_{i}^{0} = 1$ , so $P (i) = 1/4 = 0.25$ for all $i$ . Why this step? Anything to the power $0$ is $1$ — the priorities vanish, and we recover uniform DQN replay. This is cell E, the "no prioritisation" limit.
$α = 1$ (fully greedy): $P = p / \sum p = (1.51, 0.31, 0.31, 0.91) /3.04 \approx (0.497, 0.102, 0.102, 0.299)$ . Why this step? At $α = 1$ the raw priority is the (unnormalised) probability — the sharpest, most greedy focus on high-error transitions. This is cell F.

Verify: $α = 0$ : $P = 0.25$ each, sum $= 1$ . ✓ $α = 1$ : sum $= 1$ , and the top error's share rose from $0.401$ (at $α = 0.6$ ) to $0.497$ — monotonically sharper as $α$ grows, as expected. ✓

Worked example Example 6 — Cell G: importance-sampling weights + max-normalisation

Using the $α = 0.6$ probabilities $P = (0.401, 0.152, 0.152, 0.295)$ , buffer size $N = 4$ , and $β = 0.5$ . Compute the IS weights $w_{i}$ , then normalise by their max.

Forecast: Which transition gets the smallest weight — the one sampled most or least often?

Raw weight. $w_{i} = (\frac{1}{N} \cdot \frac{1}{P ( i )})^{β} = (\frac{1}{4 P ( i )})^{0.5}$ .
- $w_{1} = (1/ (4 \cdot 0.401))^{0.5} = (0.6234)^{0.5} \approx 0.7896$
- $w_{2} = (1/ (4 \cdot 0.152))^{0.5} = (1.6447)^{0.5} \approx 1.2825$
- $w_{3} \approx 1.2825$ (same as $w_{2}$ )
- $w_{4} = (1/ (4 \cdot 0.295))^{0.5} = (0.8475)^{0.5} \approx 0.9206$ Why this step? The over-sampled transition ( $P_{1}$ largest) gets the smallest weight — we down-weight it to undo the bias of picking it too often. That's the whole point of Importance Sampling.
Normalise by the max. $max_{j} w_{j} = w_{2} = 1.2825$ . Divide all: $\tilde{w} \approx (0.6157, 1.0, 1.0, 0.7178)$ . Why this step? Dividing by $max$ keeps every weight $\leq 1$ , so the correction can only shrink gradients — never blow them up. This is a stability trick, not a change of relative sizes.

Verify: The most-sampled transition ( $e_{1}$ ) got the smallest normalised weight ( $0.616$ ) — forecast confirmed. All $\tilde{w} \leq 1$ . ✓ (As $β \to 1$ the weights would spread further, fully correcting the bias.)

Worked example Example 7 — Cell H: limiting behaviour of buffer size on correlation

A car game emits states $3$ pixels apart. Buffer $D$ holds the last $N$ transitions. Sample two at random. Estimate the chance they are temporally adjacent (within, say, 5 steps of each other) for $N = 10$ vs $N = 1000$ .

Forecast: As $N$ grows $100 \times$ , does adjacency probability go up or down?

Model adjacency. For a random pick, the chance the second pick lands within a fixed window of $\pm 5$ steps around the first is $\approx \frac{10}{N}$ (window size over pool size). Why this step? Adjacent samples are the correlated ones SGD hates — the parent's "Disease 1". We want this small.
Plug in. $N = 10$ : $\approx 10/10 = 1.0$ (basically certain to be near each other). $N = 1000$ : $\approx 10/1000 = 0.01$ . Why this step? This is the quantitative statement of "bigger buffer ⇒ more i.i.d.-like samples".

Verify: $10/1000 = 0.01 ≪ 1.0 = 10/10$ . Larger buffer ⇒ far lower correlation. ✓ Matches the parent's Forecast answer that correlation goes down with $100 \times$ size. (Beware: too large $N$ keeps stale data — the freshness trade-off.)

Worked example Example 8 — Cell I: real-world word problem, end to end

A trading bot uses DQN with replay. It just made a rare, profitable trade: $r = + 10$ , non-terminal, $γ = 0.99$ , target-net best future value $max Q_{θ^{-}} = 20$ . Current estimate $Q_{θ} (s, a) = 15$ . (a) What's $δ$ ? (b) Under PER with $α = 1, ϵ = 0$ , if the only other buffered priority is $0.9$ , what fraction of draws hit this trade?

Forecast: A rare high-reward event — should PER sample it more or less than average?

Target and error. $y = 10 + 0.99 \cdot 20 = 10 + 19.8 = 29.8$ ; $δ = 29.8 - 15 = 14.8$ . Why this step? Huge positive $δ$ ⇒ the bot badly under-valued a lucrative move — precisely the "rare informative experience" the parent said replay must not throw away.
Priority. $p = ∣ δ ∣ + 0 = 14.8$ . Other transition $p^{'} = 0.9$ .
PER probability ( $α = 1$ ). $P (trade) = \frac{14.8}{14.8 + 0.9} = \frac{14.8}{15.7} \approx 0.9427$ . Why this step? PER routes ~94% of compute to the transition it's most wrong about — the rare win gets rehearsed dozens of times instead of once.

Verify: $y = 29.8$ , $δ = 14.8$ , $P \approx 0.9427$ , and $0.9427 + (0.9/15.7) = 0.9427 + 0.0573 = 1$ . ✓ Rare high-value event is sampled far more — forecast confirmed. Compare DDPG / Soft Actor-Critic, which use the same buffer trick for continuous actions.

Worked example Example 9 — Cell J: exam twist, off-policy validity

Exam question: "You bolt a replay buffer onto vanilla policy gradient (an on-policy method). After training, performance is worse than without replay. Explain quantitatively why, using an importance ratio."

Forecast: Is stored data from an old policy usable as-is to estimate a gradient for the current policy?

Name the distributions. A stored transition was generated by an old policy $π_{old}$ ; policy gradient needs an expectation under the current $π_{θ}$ . Why this step? Off-policy vs On-policy: on-policy estimators are only unbiased for data from the policy being improved.
The correction that's missing. The unbiased fix is an importance ratio $ρ = \frac{π _{θ} ( a ∣ s )}{π _{old} ( a ∣ s )}$ . Suppose $π_{θ} (a ∣ s) = 0.1$ but $π_{old} (a ∣ s) = 0.8$ . Then $ρ = 0.1/0.8 = 0.125$ . Why this step? Using raw replayed samples silently assumes $ρ = 1$ . Here the true weight is $0.125$ — an $8 \times$ over-count of that action's gradient.
Conclusion. Without multiplying by $ρ$ , the gradient is biased by up to that factor, so training degrades. DQN dodges this because its Bellman target is off-policy by construction (the $max$ makes it independent of the behaviour policy).

Verify: $ρ = 0.1/0.8 = 0.125 \neq = 1$ . Non-unit ratio ⇒ raw replay is biased for on-policy PG. ✓ This is the parent's "replay isn't universal" mistake, made quantitative.

Recall Self-test across the matrix

Terminal flag $d = 1$ makes the target equal to what? ::: Just the reward $r$ (the future term is zeroed by $(1 - d)$ ). At $α = 0$ , every PER sampling probability equals? ::: $1/ N$ — plain uniform replay. The IS weight for the most-sampled transition is the largest or smallest? ::: Smallest (we down-weight over-sampled data). Why divide IS weights by $max_{j} w_{j}$ ? ::: To keep all weights $\leq 1$ so the correction only shrinks gradients (stability). Doubling buffer size does what to correlation between two random samples? ::: Roughly halves it — more i.i.d.-like.

Worked examples — Experience replay

The scenario matrix

Worked examples