Double DQN and Dueling DQN: Exercises — AI-ML deep dive

Two boxed formulas power almost everything below. We restate them so you never have to hunt:

Level 1 — Recognition

Problem 1.1

In one sentence each, name which disease of DQN each variant cures: Double DQN and Dueling DQN.

Recall Solution

Double DQN cures overestimation bias — the optimistic upward tilt caused by taking a $max$ over noisy estimates.
Dueling DQN cures sample inefficiency — it stops the net from re-learning "how good is this state?" separately for every action, by splitting off a shared value stream. Neither fix contradicts the other; they stack (see Problem 4.1).

Problem 1.2

Circle the network that does selection and the one that does evaluation in the Double DQN target. Which network is frozen?

Recall Solution

Selection (the $ar g max$ ): the online network $θ$ .
Evaluation (reading the value at the chosen action): the target network $θ^{-}$ .
The frozen one is the target network $θ^{-}$ — a slowly-updated copy, see Target Networks and Experience Replay. The online net $θ$ updates every gradient step. Mnemonic from the parent: "Select-Online, Eval-Target" (SO-ET).

Level 2 — Application

Problem 2.1

At next state $s^{'}$ you have $r = 1$ , $γ = 0.9$ . The two networks output, over 3 actions:

Online $θ$ : $Q_{θ} (s^{'}, \cdot) = [+ 0.3, - 0.5, + 0.1]$
Target $θ^{-}$ : $Q_{θ^{-}} (s^{'}, \cdot) = [- 0.2, + 0.4, - 0.1]$

Compute $y^{D QN}$ and $y^{D o u b l eD QN}$ .

Recall Solution

Plain DQN. Select and evaluate with $θ^{-}$ . $max_{a} Q_{θ^{-}} = + 0.4$ (action $a_{2}$ ). $y^{D QN} = 1 + 0.9 \times 0.4 = 1 + 0.36 = 1.36$ Double DQN. Select with online: $ar g max_{a} Q_{θ} = a_{1}$ (value $+ 0.3$ ). Evaluate that same $a_{1}$ with $θ^{-}$ : $Q_{θ^{-}} (a_{1}) = - 0.2$ . $y^{D o u b l eD QN} = 1 + 0.9 \times (- 0.2) = 1 - 0.18 = 0.82$ Why Double gives the lower (less inflated) target: the true action values here are all $0$ , so the "right" bootstrap value is near $0$ . Plain DQN latched onto $+ 0.4$ (whichever action got lucky noise in the same net that scored it). Double DQN's evaluator $θ^{-}$ has noise independent of $θ$ 's luck on $a_{1}$ , so it does not confirm the optimism.

Problem 2.2

A Dueling head outputs $V (s) = 5.0$ and raw advantages $\tilde{A} = [+ 3, + 1, + 2]$ for 3 actions. Using mean-centering, compute $Q (s, \cdot)$ and verify $V = mean (Q)$ .

Recall Solution

Mean of $\tilde{A} = (3 + 1 + 2) /3 = 2$ . Centered advantages: $A = [3 - 2, 1 - 2, 2 - 2] = [+ 1, - 1, 0]$ Recombine: $Q = 5 + [1, - 1, 0] = [6, 4, 5]$ . Check the anchor: $mean (Q) = (6 + 4 + 5) /3 = 5 = V$ ✔. Why centering matters here: if we had skipped it and used $Q = V + \tilde{A} = [8, 6, 7]$ , then $V = 5$ would no longer be the average of $Q$ (that average is $7$ ). The $+ 2$ offset would have leaked into $Q$ , silently inflating everything and making $V$ meaningless.

Level 3 — Analysis

Problem 3.1

Prove that when the online and target networks are identical ( $θ = θ^{-}$ ), the Double DQN target equals the plain DQN target. What does this tell you about early training?

Recall Solution

If $θ = θ^{-}$ then $Q_{θ} \equiv Q_{θ^{-}}$ , so the selection $ar g max_{a^{'}} Q_{θ} (s^{'}, a^{'})$ picks the same action as $ar g max_{a^{'}} Q_{θ^{-}} (s^{'}, a^{'})$ , and evaluating with $θ^{-}$ reads the same number. Hence $y^{D o u b l eD QN} = r + γ Q_{θ^{-}} (s^{'}, ar g max Q_{θ^{-}}) = y^{D QN} .$ Interpretation: right after a target-network sync (when $θ^{-} \leftarrow θ$ ), Double DQN offers no correction — the two nets agree. The de-biasing power comes from the drift between $θ$ and $θ^{-}$ that builds up between syncs. Their estimation noises decorrelate as $θ$ keeps training while $θ^{-}$ stays frozen.

Problem 3.2

Using the bound from the parent note, $E [max_{a} ε_{a}] \leq σ \frac{m - 1}{2 m - 1}$ for $m$ i.i.d. zero-mean estimation errors of standard deviation $σ$ : with $σ = 1$ , evaluate the bound for $m = 1, 2, 5, 10$ and describe the trend.

Recall Solution

Plug in ( $σ = 1$ ):

$m = 1 : \frac{0}{1} = 0$
$m = 2 : \frac{1}{3} \approx 0.5774$
$m = 5 : \frac{4}{9} = \frac{4}{3} \approx 1.3333$
$m = 10 : \frac{9}{19} \approx 2.0647$

Trend: the upper bound on overestimation grows monotonically with the number of actions $m$ . With one action there is nothing to max over so the bias is exactly $0$ . With more actions there are more chances for one to get lucky noise, so the max drifts further above the true (zero) mean. Consequence: Double DQN's benefit is largest in large action spaces — exactly where vanilla DQN suffers most from Overestimation Bias in Q-learning.

Level 4 — Synthesis

Problem 4.1

You build a Double Dueling DQN. State (a) which part changes the network architecture, (b) which changes the target computation, and (c) whether these two changes ever conflict.

Recall Solution

(a) Architecture: the Dueling part — split the trunk into a scalar value stream $V_{η} (s)$ and a vector advantage stream $A_{ψ} (s, a)$ , recombined with mean-centering. This changes only how $Q$ is produced from features.
(b) Target computation: the Double part — the label $y$ used in the loss becomes $y^{D o u b l eD QN}$ (select-online, eval-target). This changes only what number we regress toward.
(c) Conflict? None. They are orthogonal: one edits the forward pass, the other edits the training target. Whatever $Q$ -values the dueling head produces are simply fed into the Double DQN target rule. This is why "Double Dueling DQN" is a standard baseline and a component of Rainbow DQN.

Problem 4.2

A Dueling head must output the same final $Q = [6, 4, 5]$ regardless of whether it internally chose $V = 5, \tilde{A} = [1, - 1, 0]$ or $V = 100, \tilde{A} = [96, 94, 95]$ . Show both raw pairs give $Q = [6, 4, 5]$ after mean-centering, and explain what this "sameness" is telling you.

Recall Solution

Pair 1: $\tilde{A} = [1, - 1, 0]$ , mean $= 0$ , centered $A = [1, - 1, 0]$ , $Q = 5 + [1, - 1, 0] = [6, 4, 5]$ ✔. Pair 2: $\tilde{A} = [96, 94, 95]$ , mean $= 95$ , centered $A = [1, - 1, 0]$ , $Q = 100 + [1, - 1, 0] = [101, 99, 100]$ .

These are not equal! Pair 2 gives $[101, 99, 100]$ , not $[6, 4, 5]$ . The trick: to force equality we'd also need $V$ to absorb the mean. The identifiability point is the reverse: many raw $(V, \tilde{A})$ pairs collapse to the same centered $A$ , and mean-centering pins $V$ to the true average of $Q$ . So for Pair 2 the centered advantages match Pair 1 ( $[1, - 1, 0]$ ) but the value $V$ differs, giving a shifted $Q$ . Lesson: centering fixes the shape of the advantage (the anchor $\sum_{a} A = 0$ ) but the value stream still sets the overall level — that is exactly the clean, unique split we wanted, with $V = mean (Q)$ in each case ( $5$ then $100$ ).

Level 5 — Mastery

Problem 5.1

Full pipeline. Transition: $r = 2$ , $γ = 0.99$ , $∣ A ∣ = 3$ . At $s^{'}$ a Dueling net gives:

Online $θ$ : $V_{θ} = 10$ , raw $\tilde{A}_{θ} = [+ 2, + 0, - 1]$
Target $θ^{-}$ : $V_{θ^{-}} = 8$ , raw $\tilde{A}_{θ^{-}} = [- 1, + 2, + 0]$

Compute the mean-centered $Q_{θ} (s^{'}, \cdot)$ and $Q_{θ^{-}} (s^{'}, \cdot)$ , then produce the Double DQN target. Contrast with the plain DQN target.

Recall Solution

Step 1 — center online advantages. mean $\tilde{A}_{θ} = (2 + 0 - 1) /3 = 1/3$ . $A_{θ} = [2 - \frac{1}{3}, 0 - \frac{1}{3}, - 1 - \frac{1}{3}] = [\frac{5}{3}, - \frac{1}{3}, - \frac{4}{3}]$ . $Q_{θ} = 10 + A_{θ} = [11.6 \overline{6}, 9.6 \overline{6}, 8.6 \overline{6}]$ . Step 2 — center target advantages. mean $\tilde{A}_{θ^{-}} = (- 1 + 2 + 0) /3 = 1/3$ . $A_{θ^{-}} = [- 1 - \frac{1}{3}, 2 - \frac{1}{3}, 0 - \frac{1}{3}] = [- \frac{4}{3}, \frac{5}{3}, - \frac{1}{3}]$ . $Q_{θ^{-}} = 8 + A_{θ^{-}} = [6.6 \overline{6}, 9.6 \overline{6}, 7.6 \overline{6}]$ . Step 3 — Double DQN. Select with online: $ar g max Q_{θ} = a_{1}$ (value $11.6 \overline{6}$ ). Evaluate $a_{1}$ with target: $Q_{θ^{-}} (a_{1}) = 6.6 \overline{6}$ . $y^{D o u b l eD QN} = 2 + 0.99 \times 6.6 \overline{6} = 2 + 6.6 = 8.6$ Step 4 — plain DQN for contrast. $max Q_{θ^{-}} = a_{2}$ (value $9.6 \overline{6}$ ). $y^{D QN} = 2 + 0.99 \times 9.6 \overline{6} = 2 + 9.57 = 11.57$ Reading it: plain DQN grabs the biggest target-net number ( $a_{2}$ 's $9.6 \overline{6}$ ) — the classic optimism. Double DQN forces the choice through the online net (which prefers $a_{1}$ ), then reads $a_{1}$ under the target net ( $6.6 \overline{6}$ ), producing the noticeably lower, less-inflated target $8.6$ vs $11.57$ .

Problem 5.2

Design reasoning. The parent note mentions a max-centered alternative $Q = V + (A - max_{a^{'}} A)$ , which pins $V = max_{a} Q = V^{b es t}$ . Give one reason mean-centering is preferred, and connect $V$ to the Advantage Function / Actor-Critic Methods viewpoint.

Recall Solution

Why mean over max: subtracting the mean is a smooth, low-variance operation — a small change in any advantage nudges the anchor only by $1/∣ A ∣$ . Subtracting the max depends on which action is currently the argmax; that argmax can flip between gradient steps, injecting a discontinuous, higher-variance signal into $V$ and destabilising training. Connection: with mean-centering the constraint is $\sum_{a} A (s, a) = 0$ , i.e. advantages are relative to the average action. That matches the classical Advantage Function definition $A^{π} (s, a) = Q^{π} (s, a) - V^{π} (s)$ with $V^{π} (s) = E_{a \sim π} [Q^{π}]$ — a mean, not a max. In Actor-Critic Methods the critic learns exactly this $V$ , and the advantage tells the actor how much better than average each action is. Dueling DQN bakes that same value/advantage decomposition into the network's shape.

Recall Final self-check

Which network selects in Double DQN? ::: The online net $θ$ (evaluation is done by target $θ^{-}$ ). After a target sync $θ^{-} \leftarrow θ$ , does Double DQN differ from plain DQN? ::: No — they coincide until the nets drift apart. What does mean-centering pin $V$ to? ::: The average of $Q$ over actions, $V (s) = \frac{1}{∣ A ∣} \sum_{a} Q (s, a)$ . Are Double and Dueling in conflict? ::: No — one edits the architecture, the other the target; they stack into Double Dueling DQN.