Worked examples — Deep Q-Networks (DQN)

Before anything, let us re-earn the symbols we will use on every line.

See Bellman Equation, Q-Learning and Temporal Difference Learning for where these come from.

---

The scenario matrix

Every DQN target-computation you will ever do lives in one of these cells. We will hit all of them.

Cell	Case class	What is special	Example
A	Normal non-terminal, positive reward	the "textbook" case	Ex 1
B	Terminal state	drop the $\gamma \max$ term, $y=r$	Ex 2
C	Negative reward	sign flips the target	Ex 3
D	$\gamma =0$ (myopic) vs $\gamma \to 1$ (far-sighted)	limiting values of discount	Ex 4
E	Tie in the $\max$	two actions equally best	Ex 5
F	Minibatch (several transitions at once)	vector of targets & mean loss	Ex 6
G	Real-world word problem	translate a story into $(s,a,r,s^{\prime })$	Ex 7
H	Exam twist — backprop-through-target trap	conceptual gradient check	Ex 8
I	Overestimation & the Double-DQN fix	why $\max$ is biased	Ex 9

A small map of how a single transition flows into a number. Watch which network each box uses:

---

Example 1 — Cell A: normal non-terminal, positive reward

Forecast: Guess now — will $\delta$ be positive or negative? Is our guess of $5$ too low?

1. Take the max over next actions: ${\max }_{a^{\prime }}Q(s^{\prime },a^{\prime })=\max (4,7,1)=7$. Why this step? Bellman assumes we will act optimally from $s^{\prime }$ onward, so we credit the best available next value, not the average.
2. Build the target: $y=r+\gamma \cdot 7=2+0.9(7)=2+6.3=8.3$. Why this step? Split reward into "now" ($r$) plus "discounted best future" — this is the bootstrapped stand-in for the true $Q^{\ast }$ we cannot see.
3. TD error: $\delta =y-Q(s,a;\theta )=8.3-5=3.3$. Why this step? $\delta$ tells the gradient step which way and how hard to nudge the guess.

Verify: $\delta >0$, matching the forecast that $5$ was too low; the update raises $Q(s,a)$ toward $8.3$. Units: everything is in "reward" units, consistent.

---

Example 2 — Cell B: terminal state

Forecast: The future is gone. Will $y$ be bigger or smaller than in Example 1?

1. Recognise termination: there is no $s^{\prime }$ to evaluate, so the $\gamma \max Q$ term vanishes. Why this step? No actions happen after the episode ends, so there is no future reward to discount.
2. Target: $y=r=2$. Why this step? The definition of $Q$ is the total future reward, which here is just $r$.
3. TD error: $\delta =2-5=-3$. Why this step? Now the guess $5$ is too high, so we push it down.

Verify: Compare to Example 1 ($y=8.3$): dropping the future term shrinks the target from $8.3$ to $2$, exactly as expected. Sign of $\delta$ flipped from $+$ to $-$: same state-guess, but the correct value collapsed because nothing follows.

---

Example 3 — Cell C: negative reward

Forecast: A big negative reward — will $y$ end up negative?

1. Max over next actions: $\max (3,-2)=3$. Why this step? Even after a penalty, the best future option still governs the bootstrap.
2. Target: $y=-5+0.8(3)=-5+2.4=-2.6$. Why this step? Reward is negative but the future is mildly positive; they partly cancel.
3. TD error: $\delta =-2.6-0=-2.6$. Why this step? Guess of $0$ was optimistic; push it down toward $-2.6$.

Verify: $y$ is negative but not as negative as $r=-5$, because a positive future softened it — sanity-consistent with "penalty now, some hope later". $\delta <0$ correctly says "lower it".

---

Example 4 — Cell D: the two extremes of $\gamma$

The discount $\gamma$ is a dial. To feel it, we hold everything else fixed and turn the dial to both ends. This is the "limiting behaviour" row of the matrix.

What to see in the figure: the red line is the target $y=1+10\gamma$, with the horizontal axis labelled discount factor gamma (from $0$ to $1$) and the vertical axis labelled target y. Three black dots sit exactly on the line at $(\gamma ,y)=(0.00,1.00)$, $(0.50,6.00)$ and $(0.99,10.90)$ — each dot is annotated with its own $(\gamma ,y)$ pair so you can read the answer straight off the picture. The red line starts at the immediate reward $1$ (where $\gamma =0$ kills the future) and would reach $11$ at $\gamma =1$ (full future value $10$ added).

Forecast: Which $\gamma$ makes the far-future value of $10$ matter most?

1. $\gamma =0$ (myopic): $y=1+0\cdot 10=1$. Why this step? $\gamma =0$ zeroes the future entirely — the agent only ever sees the next reward.
2. $\gamma =0.5$: $y=1+0.5\cdot 10=6$. Why this step? Future is worth half; the big $10$ contributes $5$.
3. $\gamma =0.99$ (far-sighted): $y=1+0.99\cdot 10=10.9$. Why this step? Nearly all of the future value flows through; the target is dominated by it.

Verify: As $\gamma$ rises $0\to 0.5\to 0.99$, $y$ rises $1\to 6\to 10.9$ — monotone, matching the rising red line and the three annotated black dots in the figure (heights $1$, $6$, $10.9$).

---

Example 5 — Cell E: a tie in the $\max$

Forecast: Does a tie break the formula? What number does $\max$ return?

1. Max ignores which action wins: $\max (4,4,-1)=4$. Why this step? $\max$ returns the value, not the identity of the action. Ties are fine for $y$; the tie only matters if you also need $\arg \max$ (for acting), where you'd pick either.
2. Target: $y=0+1\cdot 4=4$. Why this step? Undiscounted ($\gamma =1$), zero immediate reward, so $y$ is purely the future.

Verify: Swapping which tied action we "prefer" leaves $y=4$ unchanged — the target is well-defined even under ties. (Contrast: $ϵ$-greedy acting would break the tie randomly; see Epsilon-Greedy Exploration.)

---

Example 6 — Cell F: a full minibatch

Real DQN trains on a minibatch: several transitions sampled from Experience Replay at once. The loss is the mean squared TD error over the batch.

Forecast: Which transition contributes the biggest squared error?

1. $y_1=1+0.9(5)=5.5$; ${\delta }_1=5.5-3=2.5$. Why? Cell-A style (non-terminal).
2. $y_2=r=2$; ${\delta }_2=2-2=0$. Why? Terminal — drop the future term (Cell B). This sample is already perfect, contributes nothing.
3. $y_3=0+0.9(10)=9$; ${\delta }_3=9-6=3$. Why? Non-terminal, zero immediate reward.
4. Mean loss: $\mathcal{L}=\frac{1}{3}(2.5^2+0^2+3^2)=\frac{1}{3}(6.25+0+9)=\frac{15.25}{3}\approx 5.0833$. Why this step? Averaging (not summing) keeps the gradient scale independent of batch size.

Verify: Transition 3 has the largest $|\delta |=3$, so it dominates the loss — matches forecast. Terminal transition 2 correctly adds $0$. $\mathcal{L}\approx 5.083>0$, as any squared error must be.

---

Example 7 — Cell G: real-world word problem

Forecast: Immediate reward is negative, but the dock has a great "scan" option. Net effect?

1. Translate the story: $s=$ aisle $A$, $a=$ forward, $r=-1$, $s^{\prime }=$ dock (non-terminal), next values $\{8,3,-4\}$. Why this step? DQN only ever sees the tuple $(s,a,r,s^{\prime })$; the story must become numbers.
2. Max over next actions: $\max (8,3,-4)=8$. Why? The dock's best follow-up is "scan".
3. Target: $y=-1+0.95(8)=-1+7.6=6.6$. Why? Time cost now, big scan value soon.
4. TD error: $\delta =6.6-5=1.6>0$. Why? The move led somewhere valuable; upgrade the guess.

Verify: Despite the negative reward, moving forward is judged better than the old guess of $5$ because it unlocked a high-value next state — a nice reminder that a penalising step can still be optimal. Units: all in reward units; $\delta$ positive ⇒ "raise $Q(\text{aisle }A,\text{forward})$".

---

Example 8 — Cell H: exam twist, the backprop-through-target trap

Forecast: If we do let gradients flow into $y$, is the target a fixed post or a moving one?

1. Correct target is detached: in real DQN $y$ uses ${\theta }^{-}$ (frozen), so ${\nabla }_{\theta }y=0$. Why this step? We are solving a fixed-point equation $Q=\mathcal{T}Q$; the right side is treated as a label, like $y$-values in supervised learning.
2. Correct per-sample gradient: ${\nabla }_{\theta }\mathcal{L}=-2(y-Q){\nabla }_{\theta }Q$. With $y=1+0.9(5)=5.5$, $Q=3$: ${\nabla }_{\theta }\mathcal{L}=-2(5.5-3)(1)=-5$. Why this step? Only $Q(s,a;\theta )$ carries the gradient; $y$ is a constant.
3. Update size: with step $\alpha$, $\theta \leftarrow \theta -\alpha (-5)=\theta +5\alpha$, raising $Q(s,a)$ toward the fixed $5.5$. Why? Chases a still target ⇒ stable descent.
4. The trap: if instead ${\nabla }_{\theta }y\ne 0$, the "target" shifts every step (dog-chasing-tail), converting a stable regression into an unstable dynamical system that can diverge — the very reason Double DQN and the target-network trick exist.

Verify: Correct gradient $=-5$; the update raises the guess from $3$ toward $5.5$, and the sign ($-5$ times $-\alpha$ = up) matches the positive TD error $\delta =2.5$. The exam-correct answer is "detach $y$" — code word stop_gradient / .detach().

---

Example 9 — Cell I: overestimation and the Double-DQN fix

DQN's $\max$ is biased upward whenever the value estimates are noisy: taking the max of noisy numbers tends to pick a positive error. This is why Double DQN splits choosing from evaluating.

What to see in the figure: the true next-state value is $0$ (the black dotted line). The red bar is DQN's $\max$ estimate — it sits above the truth at $+0.6$ because the max latched onto the luckiest positive noise. The right-hand black bar is Double-DQN's estimate at $-0.1$, which hugs the dotted truth line far more closely. The picture is the whole lesson: red overshoots, Double-DQN corrects.

Forecast: The truth is $0$. Which of the two targets do you expect to land closer to $0$?

1. Vanilla DQN — max of noisy values: $\max (0.6,-0.5,0.2)=0.6$, so $y_{\text{DQN}}=0+1(0.6)=0.6$. Why this step? $\max$ selects whichever action got the luckiest positive noise, so it systematically overshoots the true $0$ — this is the overestimation bias.
2. Double-DQN — step 1, choose: the online net's $\arg \max$ over $(0.6,-0.5,0.2)$ is action $1$ (value $0.6$ is largest). Why this step? One network decides which action looks best; call it $a^{\star }=1$.
3. Double-DQN — step 2, evaluate with a different net: the evaluator net scores that same action $1$ as $-0.1$, so $y_{\text{DDQN}}=0+1(-0.1)=-0.1$. Why this step? Because a separate network supplies the value, the positive noise that inflated the pick does not also inflate the score — the two noises are independent, so the upward bias cancels on average.
4. Compare both to truth $0$: $|y_{\text{DQN}}-0|=0.6$ versus $|y_{\text{DDQN}}-0|=0.1$. Why this step? Smaller absolute error ⇒ less bias; Double-DQN is the better target here.

Verify: $y_{\text{DDQN}}=-0.1$ is closer to the true $0$ than $y_{\text{DQN}}=0.6$ ($0.1<0.6$), illustrating the overestimation fix, and matching the figure where the red bar overshoots to $+0.6$ while the Double-DQN bar sits near the dotted truth line at $-0.1$.

---

Recall Quick self-test across the matrix

Non-terminal, $r=3$, $\gamma =0.5$, $\max Q(s^{\prime })=8$: what is $y$? ::: $y=3+0.5(8)=7$. Same but terminal: what is $y$? ::: $y=r=3$. Why does a tie in $\max$ not break the target? ::: $\max$ returns the value, not the action identity. In a minibatch, do we sum or average the squared TD errors? ::: Average, to keep gradient scale batch-size-independent. Why detach the target $y$? ::: It is a fixed label; differentiating through it chases a moving target and can diverge. Why is plain DQN's $\max$ overoptimistic? ::: Taking the max of noisy estimates systematically selects positive errors. How does Double-DQN reduce that bias? ::: One net chooses the action, a separate net scores it, so the selection noise does not also inflate the value.

---

Connections

• Deep Q-Networks (DQN) — the parent note these examples drill.
• Bellman Equation — every target here is one Bellman backup.
• Temporal Difference Learning — $\delta$ is the TD error.
• Experience Replay — where Example 6's minibatch comes from.
• Double DQN — the fix demonstrated in Example 9.
• Epsilon-Greedy Exploration — how ties (Example 5) are broken while acting.