Deep Q-Networks (DQN): Exercises — AI-ML deep dive

A quick reminder of every symbol that appears below, in plain words:

Definition Symbols you will keep using on this page

$Q (s, a; θ)$ — the network's current guess of "how good is action $a$ in state $s$ ". Think of it as a number the brain-network prints when you show it a screen and a button.
$y = r + γ max_{a^{'}} Q (s^{'}, a^{'}; θ^{-})$ — the target: the reward you just got, $r$ , plus a discounted ( $γ$ , a number between 0 and 1 that shrinks future rewards) estimate of the best you can do from the next screen $s^{'}$ , computed by the frozen copy $θ^{-}$ .
The TD error is $δ = y - Q (s, a; θ)$ : how wrong the current guess is.
$D$ — the replay buffer: a notebook that stores past transitions $(s, a, r, s^{'})$ so we can sample old experiences at random to train on. Introduced in Experience Replay.
$C$ — the target-network refresh interval: the number of gradient steps we wait before copying the online weights into the frozen target, $θ^{-} \leftarrow θ$ .

The figures on this page are visual-first: each core intuition (Bellman update, replay shuffling, moving-vs-frozen target) gets its own picture so you can see the idea before the algebra.

L1 — Recognition

Problem 1.1

Which component of DQN is responsible for breaking correlation between consecutive frames? (a) target network, (b) $ϵ$ -greedy, (c) experience replay, (d) discount factor.

Figure 1 shows the intuition: raw play produces a correlated stream; the buffer $D$ shuffles it into a scrambled minibatch.

Recall Solution

(c) experience replay. Why: consecutive frames of a game look almost identical, so training on them in order feeds the optimiser near-duplicate, non-independent samples. Sampling a random minibatch from the buffer $D$ (defined above: the notebook of past transitions) shuffles far-apart moments together, restoring the "independent samples" assumption that gradient descent relies on. The target network (a) fixes a different problem — a moving target. See Experience Replay.

Problem 1.2

For a terminal transition, what is the target $y$ ?

Recall Solution

$y = r$ . Why: "terminal" means the episode ended, so there is no next state $s^{'}$ to look forward from — the $γ max_{a^{'}} Q (s^{'}, a^{'})$ term has nothing to evaluate and is dropped.

L2 — Application

Problem 2.1

Non-terminal transition. $r = 2$ , $γ = 0.9$ . The target network outputs, for the next state $s^{'}$ : $Q (s^{'}, up) = 4$ , $Q (s^{'}, down) = 7$ , $Q (s^{'}, stay) = 1$ . The current network says $Q (s, a; θ) = 5$ . Compute the target $y$ and the TD error $δ$ .

Figure 2 pictures this one update: pick the best next value, discount it, add the reward, then compare to the current guess.

Recall Solution

Step 1 — best next action. $max_{a^{'}} Q (s^{'}, a^{'}) = max (4, 7, 1) = 7$ . Why: Bellman assumes we act optimally from $s^{'}$ onward, so we take the largest value. Step 2 — target. $y = r + γ \cdot 7 = 2 + 0.9 (7) = 2 + 6.3 = 8.3$ . Step 3 — TD error. $δ = y - Q (s, a; θ) = 8.3 - 5 = 3.3$ . Positive $δ$ means the net underestimates; the update nudges $Q (s, a)$ up toward $8.3$ .

Problem 2.2

Same numbers as 2.1, but now $s^{'}$ is terminal. Recompute $y$ and $δ$ .

Recall Solution

$y = r = 2$ (drop the future term). $δ = 2 - 5 = - 3$ . Negative $δ$ ⇒ the net overestimates here; update pushes $Q (s, a)$ down.

Problem 2.3

One SGD step. With $δ = 3.3$ from Problem 2.1, learning rate $α = 0.1$ , and $\nabla_{θ} Q (s, a; θ)$ pointing so that a scalar update $Q_{new} = Q_{old} + α δ$ approximates the effect on the single output, what is the updated estimate $Q_{new} (s, a)$ ?

Recall Solution

$Q_{new} = 5 + 0.1 (3.3) = 5 + 0.33 = 5.33$ . Why: one gradient step moves the output a fraction $α$ of the way toward the target; it does not jump all the way to $8.3$ , which keeps learning stable.

L3 — Analysis

Problem 3.1

Explain, in terms of a fixed point, why we do not backpropagate through the target $y$ even though $y$ contains a $Q$ value.

Recall Solution

We want $Q (s, a; θ)$ to equal its Bellman target — that is a fixed-point condition: "left side = a function of the right side." We solve it like a regression: freeze the right side as a label $y$ and pull the left side toward it. If instead we differentiated through $y$ , the optimiser would also move the label as it moves the prediction — it minimises the gap by dragging the target down toward the prediction rather than lifting the prediction to the true value. That collapses the fixed-point equation into an unstable feedback loop and diverges. This is exactly why $y$ uses the frozen $θ^{-}$ and is .detach()-ed. See Bellman Equation and Temporal Difference Learning.

Problem 3.2

The plot below (Figure 3) shows a scalar toy where the target is $y = 0.9 Q + 1$ (a Bellman-like update) and the true fixed point is where $Q = y$ . Trace what "chasing a moving target" looks like versus a frozen target, and state which one converges.

Recall Solution

The true fixed point solves $Q = 0.9 Q + 1 \Rightarrow 0.1 Q = 1 \Rightarrow Q^{⋆} = 10$ (the mint dot where the line $y = 0.9 Q + 1$ crosses the diagonal $y = Q$ ).

Frozen target (lavender path): hold $y$ fixed for several steps, take small steps toward it, then refresh $y$ . Each staircase segment is stable and the sequence climbs steadily to $Q^{⋆} = 10$ .
Moving target (coral path): recompute $y$ every step while also moving $Q$ . The label slides as you approach, producing the overshooting / oscillating behaviour the figure shows. The frozen scheme converges reliably — this is the geometric reason the target network exists.

L4 — Synthesis

Problem 4.1 — Double DQN vs DQN target

In vanilla DQN the target is $y = r + γ max_{a^{'}} Q (s^{'}, a^{'}; θ^{-})$ . Suppose the online network's values for $s^{'}$ are $Q (s^{'}, up; θ) = 6$ , $Q (s^{'}, down; θ) = 5$ , and the target network's are $Q (s^{'}, up; θ^{-}) = 4$ , $Q (s^{'}, down; θ^{-}) = 8$ , with $r = 0$ , $γ = 1$ . (a) Compute the vanilla-DQN target. (b) Compute the Double DQN target, which uses the online net to pick the action and the target net to evaluate it: $y = r + γ Q (s^{'}, ar g max_{a^{'}} Q (s^{'}, a^{'}; θ); θ^{-})$ . (c) Which is smaller, and why does that reduce overestimation?

Recall Solution

(a) Vanilla: $max (4, 8) = 8$ , so $y = 0 + 1 \cdot 8 = 8$ . (b) Double: the online net picks the action: $ar g max (6, 5) = up$ . Then evaluate up with the target net: $Q (s^{'}, up; θ^{-}) = 4$ . So $y = 0 + 1 \cdot 4 = 4$ . (c) The Double target ( $4$ ) is smaller than the vanilla target ( $8$ ). Vanilla DQN takes a $max$ over noisy estimates, and $max$ of noisy numbers is systematically too high (it latches onto whichever action happened to be over-estimated — here "down" with value $8$ ). Decoupling selection (online) from evaluation (target) breaks that coupling: the action the online net believes is best is scored by an independent network, so the inflated $8$ is not automatically picked up. This is the overestimation fix behind Double DQN.

Problem 4.2 — Off-policy justification of replay

A transition $(s, a, r, s^{'})$ was collected 50,000 steps ago while acting nearly randomly ( $ϵ \approx 1$ ). Argue whether it is still valid to train on today, when $ϵ \approx 0.1$ .

Recall Solution

Valid. DQN learns $Q^{⋆}$ , the value of acting greedily, regardless of the (exploratory) policy that generated the data — that is what off-policy means. The target $y = r + γ max_{a^{'}} Q (s^{'}, a^{'}; θ^{-})$ contains a $max$ , i.e. it bakes in "act optimally next," so it never assumes the stored action came from the greedy policy. Hence old, random-ish transitions carry correct one-step reward + transition information and can be replayed. This reuse is precisely the sample efficiency of Experience Replay. (Contrast: an on-policy method like Policy Gradient Methods would be invalidated by stale data.)

L5 — Mastery

Problem 5.1 — Design the annealing schedule

You want $ϵ$ to decay linearly from $ϵ_{start} = 1.0$ to $ϵ_{end} = 0.1$ over the first $N = 1, 000, 000$ steps, then stay at $0.1$ . (a) Write $ϵ (t)$ for $t \leq N$ . (b) Evaluate $ϵ$ at $t = 250, 000$ . (c) In one sentence, why anneal at all instead of a constant $ϵ$ ?

Recall Solution

(a) Linear interpolation from $1.0$ down to $0.1$ :

= 1.0 - 0.9\cdot\frac{t}{N},\qquad t\le N,$$ and $\epsilon(t)=0.1$ for $t>N$. **(b)** $\epsilon(250000) = 1.0 - 0.9\cdot\frac{250000}{1000000} = 1.0 - 0.9(0.25) = 1.0 - 0.225 = 0.775$. **(c)** Explore heavily **early** (the net is ignorant, so try everything), then exploit the learned $Q$ **later** — a constant $\epsilon$ either explores too little at the start or wastes reward on randomness forever. See [[Epsilon-Greedy Exploration]].

Problem 5.2 — Convergence of the frozen-target update (prove the geometry)

For the scalar Bellman map $T (Q) = r + γ Q$ with $0 < γ < 1$ , show that repeatedly setting $Q_{k + 1} = T (Q_{k})$ (fully solving each frozen target) converges to a unique fixed point, and give its value for $r = 1, γ = 0.9$ . Relate this to Figure 3.

Recall Solution

Contraction. For any two guesses $Q, Q^{'}$ : $∣ T (Q) - T (Q^{'}) ∣ = ∣ γ Q - γ Q^{'} ∣ = γ ∣ Q - Q^{'} ∣.$ Since $γ < 1$ , each application of $T$ shrinks the gap by factor $γ$ — $T$ is a contraction. By the Banach fixed-point idea, iterating $Q_{k + 1} = T (Q_{k})$ pulls any starting guess geometrically toward one unique $Q^{⋆}$ . Fixed point. Set $Q^{⋆} = T (Q^{⋆}) = r + γ Q^{⋆} \Rightarrow Q^{⋆} (1 - γ) = r \Rightarrow Q^{⋆} = \frac{r}{1 - γ}$ . For $r = 1, γ = 0.9$ : $Q^{⋆} = \frac{1}{0.1} = 10$ . Link to Figure 3: $Q^{⋆} = 10$ is the mint dot where $y = 0.9 Q + 1$ meets the diagonal $y = Q$ ; the lavender staircase is exactly this contraction, each step multiplying the remaining distance to 10 by $0.9$ .

Recall One-line self-check before you leave

Non-terminal target ::: $y = r + γ max_{a^{'}} Q (s^{'}, a^{'}; θ^{-})$ Terminal target ::: $y = r$ TD error ::: $δ = y - Q (s, a; θ)$ Scalar Bellman fixed point ::: $Q^{⋆} = r / (1 - γ)$ Double DQN target ::: $y = r + γ Q (s^{'}, ar g max_{a^{'}} Q (s^{'}, a^{'}; θ); θ^{-})$

Connections

Deep Q-Networks (DQN) — the parent note these exercises drill.
Q-Learning · Bellman Equation · Temporal Difference Learning — the theory behind the targets.
Experience Replay — why old data (Problem 4.2) stays valid.
Double DQN — the overestimation fix in Problem 4.1.
Dueling DQN · Policy Gradient Methods · Epsilon-Greedy Exploration — neighbours referenced above.